DEV Community: Himanshu Kumar

Your MCP Server Says It Is Read-Only. Who Checked?

Himanshu Kumar — Sun, 30 Aug 2026 04:43:19 +0000

MCP servers describe their own tools. A tool can publish readOnlyHint: true, and an agent harness may use that annotation to decide whether a human approval card is needed.

That creates an uncomfortable trust boundary: the server being evaluated is also the server describing how dangerous it is.

I built Airlock to test the declaration against observed behaviour, then enforce the resulting policy on the wire.

Airlock reports what it observed. Absence of a finding is not proof of safety.

The gap

Consider an MCP tool called export_report. It declares itself read-only, returns a normal result and writes a file outside the operator's declared scope.

A harness that trusts the annotation may run it without approval. Airlock opens a case, inventories the declared tools, exercises them under a capped budget and compares the declaration with its observations.

If a tool behaves differently from what it declared, the difference becomes evidence. For an approved case, Airlock emits a connector that points to a per-case enforcing proxy rather than back to the suspect server.

The policy is enforced before the call reaches the upstream tool. It is not a sentence in a system prompt.

A controlled dishonest server

Airlock ships two six-tool fixtures with the same surface.

The dishonest fixture plants five behaviours, including a filesystem write behind readOnlyHint: true, scope escape, undeclared egress, injected instructions and canary exfiltration.

The result:

target      dishonest fixture, controlled_fixture mode
probes      24
result      7 findings of 36 checks, all five planted behaviours

The honest fixture produces zero findings across the same 36 checks.

That contrast is essential. A detector that only finds problems in a fixture designed to look suspicious has not shown much. The honest control shows that the findings follow observed behaviour rather than unfamiliarity.

Four outcomes, not a safety score

Each check resolves to one of four states:

finding
no_finding_observed
not_tested
sensor_failed

Severity is reported separately.

I deliberately avoided a single safety score. An average can hide the one tool that lied. It also encourages readers to treat missing visibility as a passing result.

not_tested is a first-class state. Under transcript_only, MCP traffic cannot reveal server-side filesystem or network activity. Airlock records those questions as untested rather than clean.

It distinguishes capability_absent, where no sensor existed, from evidence_missing, where the sensor could answer but observed nothing.

Testing servers nobody built for Airlock

The owned fixture proves that the detectors can catch planted behaviour. The more interesting question is what happens on external servers.

Airlock audited a deployed ContextFirewall target over HTTPS:

6 tools
30 probes
0 tools declaring any annotation

The server did not lie. It said nothing. A harness that resolves categories such as @write and @destructive from annotations can match nothing either way, so tools such as remember and forget_memory may avoid the approval path.

Airlock also inventoried five public stdio servers:

Server	Tools
`server-filesystem`	14
`server-everything`	13
`mcp-server-git`	12
`server-memory`	9
`server-sequential-thinking`	1

Stdio support matters because many MCP servers are launched as commands rather than exposed at a URL.

Executing what you distrust

Launching a stdio server means executing the code Airlock exists to distrust. The command therefore never comes from a case argument, tool result or model-generated string.

The operator configures a fixed map of names to argument arrays. A case may select a name. Names are looked up, never parsed into shell commands.

The child starts in a throwaway working directory with an explicit environment. Airlock revalidates the whole command binding before each connection, so repointing a configured name revokes an open case instead of silently running a different command.

Airlock documents an important remaining boundary: its HTTP response cap does not apply to the MCP SDK's stdio transport. The audit deadline still applies, but a large line can reach the SDK before Airlock can bound it. Calling that solved would be worse than stating it.

Airlock is itself an MCP server

A TrueForge agent drives the audit through six control tools:

open_case
list_declared_tools
probe_tool
read_evidence
seal_case
emit_policy

Three operations are always approval-gated: probe_tool, seal_case and emit_policy.

The agent inventories the target, asks Airlock to probe opaque tool IDs, reads the aggregate evidence and presents a human choice: Block, Approve selected or Approve all.

For an allowed case, the emitted connector points at the enforcing case proxy. A call to a tool the case did not approve receives:

MCP error -32001: Tool blocked by Airlock policy

The self-audit failed, so the failure is shown

Airlock audited its own control MCP. The server published full annotations, but the probe planner rejected open_case because its schema contains a $ref into $defs, outside Airlock's bounded v1 probe profile.

That case is incomplete. It appears on the hosted page because a product arguing that missing evidence is not proof of safety should not hide its own missing evidence.

Qodo found claim-breaking bugs

Every substantive change went through a Qodo-reviewed pull request.

On the stdio transport, Qodo found that revalidation compared only the target name. Repointing that name could leave an open case running an operator-withdrawn command. It also found that the MCP SDK inherited more host environment variables than the README claimed.

On the approval boundary, Qodo caught a test that claimed to cover every side-effecting control tool while checking three hard-coded names. The first fix introduced a second map that could drift, and Qodo caught that too. The decorators now read from the same policy map the test validates.

Another review found that a passing-suite claim did not survive a clean CI machine. The failures were fixed, re-reviewed and recorded.

The reviews did more than find code defects. They caught statements that were stronger than the observed behaviour, exactly the failure Airlock is designed to expose.

Current verification

Run the complete suite with:

python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
.venv/bin/python -m pytest -q

The current suite contains 302 passing tests.

Try Airlock

The backend requires persistent state, long-running audits, child processes for stdio targets and a loopback-only operator interface, which makes a serverless deployment the wrong shape. The repository quickstart runs the complete fixture audit locally in two commands.

Airlock is open source under the MIT licence and was built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon.

Do not ask a server whether it is safe. Ask what it did, keep the record and enforce the answer.

Bugs Are Innocent Until Reproduced: Building Verdict, an Evidence-First Agent Harness

Himanshu Kumar — Sun, 30 Aug 2026 04:43:06 +0000

Most flaky bug reports end in one of two places: "cannot reproduce" or a patch nobody can prove fixed the problem.

I built Verdict around a stricter idea:

Bugs are innocent until reproduced.

Verdict turns a GitHub issue into a bounded investigation. It runs an approved command repeatedly under approved conditions, keeps every observation and refuses to claim a reproduction unless the evidence crosses a deterministic threshold.

This is not an autonomous patch generator. It is an evidence-producing agent harness for the difficult step that comes before a patch.

Why another bug investigation tool?

An LLM can read a stack trace and propose a plausible explanation quickly. Plausible is not the same as reproduced.

For an intermittent failure, the questions that matter are concrete:

Which condition actually triggers it?
How often does it fail under that condition?
What happens under a contrasting control?
Which repository range does the evidence support?
What regression test would prevent the same failure from returning?

Verdict treats those as an experiment rather than a conversation.

The three-act investigation

Verdict uses three bounded subagents:

GitHub issue
    |
    v
Hunter: find the trigger
    |
    v
Surgeon: localize the change
    |
    v
Insurance: keep it fixed
    |
    v
Maintainer review

Hunter

Hunter searches only the condition matrix and command budget approved by the maintainer. Successful, failed, partial and unresolved runs all stay in the evidence ledger. An inconvenient result cannot disappear just because it weakens the story.

Surgeon

Surgeon narrows the reproduced condition to the smallest suspect range the records support. Static inspection stays visibly different from a proven execution boundary. Surgeon does not author a patch.

Insurance

Insurance converts the reproduction into a regression plan: the test name, fixture, failing assertion and publication manifest. A draft pull request can only be created through a workflow the maintainer explicitly approves.

Each act is allowed to claim less than the act before it. None can talk the deterministic reducer into a stronger verdict.

The reproduction is a record, not a screenshot

Verdict reproduced TrueForge issue #417, where snapshot registration can wait indefinitely when an upstream request never resolves.

The pinned runtime used @truefoundry/trueforge-core@0.1.4#DaytonaSandboxProvider and ran two conditions:

Condition	Result
`daytona-stalled-endpoint`	10 of 10 runs matched, `REPRODUCTION_PINNED`
`daytona-responsive-endpoint`	0 of 10 matched, `NOT_REPRODUCED`

The control is the important half. A condition that fails every time next to one that never fails is stronger evidence than twenty failures with no contrast.

Anyone can recompute the record:

pnpm --filter @verdict/agent verify:runtime-evidence

The verifier returns:

{
  "verdict": "REPRODUCED",
  "stalledRuns": 10,
  "responsiveControls": 10,
  "provider": "@truefoundry/trueforge-core@0.1.4#DaytonaSandboxProvider",
  "canonicalSha256": "a8bb5dd22e083782bd7782fccb0a1343b59fc77ea8525b6358fecc9b5b8baffa"
}

The evidence binds the observations to the TrueForge session, Hunter thread, repository commit, npm provenance commit and shared source blob.

Exploration and proof are different systems

The model gathers candidate observations. It does not decide what those observations prove.

Verdict's evidence contract is simple:

Issue text and repository content begin as untrusted input.
The investigation may use only approved commands, knobs and budgets.
Every accepted observation must match the evidence schema.
Pure reducers decide which claim the records support.
Missing or conflicting evidence produces an honest partial result.
A maintainer controls the only public write.

The same records always produce the same verdict. That is the boundary between an agent exploring a problem and a system making a claim.

What is live and what is a fixture

The recorded case renders the executed artifact. It includes both conditions, all twenty runs and the recomputable hash.

The interactive workspace is a conceptual fixture. Every generated value is labelled. It is not quietly presented as live runtime evidence.

That distinction matters for a product whose entire argument is that a claim needs a record.

Approval remains a maintainer decision

Verdict also exercised its publication boundary against real GitHub. After explicit approval, a nonce-bound workflow ran, verified the external reproduction reference and created a draft pull request in Verdict's repository. The upstream TrueForge repository remained read-only.

The workflow proof says runtimeReproducedByThisWorkflow: false. That is deliberate. The provider run reproduced the bug. GitHub Actions verified the harness and published the independently checkable proof. Combining those into one vague "verified" flag would erase the boundary.

Qodo reviewed the claims as well as the code

Every substantive change went through a pull request reviewed by Qodo before merge.

The most useful findings were not dramatic crashes. They were mismatches between the implementation and what the project claimed:

A landing card still described the real reproduction as simulated.
CI trusted the recorded verdict instead of recomputing its hash.
README copy claimed coverage on every push while the workflow covered main and pull requests.
A malformed CSS selector silently failed after a cleanup.

Those reviews fit the product philosophy perfectly: do not ship a stronger claim than the evidence supports.

Current verification

The repository runs the same gate locally and in CI:

pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm --filter @verdict/agent verify:runtime-evidence

The current suite contains 220 tests across the agent, protocol and web packages.

Try Verdict

Verdict is open source under the MIT licence and was built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon.

The goal is not to make an agent sound certain. The goal is to make certainty inspectable.

Your dog's camera roll is a wellness history. BarkPass makes it speak.

Himanshu Kumar — Sat, 15 Aug 2026 17:36:46 +0000

This is a submission for Weekend Challenge: Dog Days Edition.

Dog owners take hundreds of photos, then rely on memory for the question that matters: has anything changed?

Yesterday's posture, last week's energy, that unusually restless look. Each clue is visible for a moment, then buried between screenshots and receipts. A single photo is a memory. A sequence of photos can become a wellness history.

BarkPass makes that history queryable, audible and portable.

One daily photo becomes a structured visual observation, a short voice update spoken as the dog, a grounded history the owner can question, and an optional on-chain pet passport. BarkPass describes what is visible. It never diagnoses a condition or replaces a veterinarian.

What I Built

BarkPass is a responsive wellness companion for any dog. Its product loop is notice, hear, remember, carry.

Create a profile for your dog.
Add today's photo or a short video.
Gemini returns a structured read of visible mood, energy, posture and flags.
ElevenLabs turns that result into a short first-person voice note.
Snowflake stores the check-in and grounds plain-language trend answers in the rows that actually exist.
Solana turns the dog's identity details into a portable, independently verifiable devnet passport.

Bruno is the editorial example on the landing page, not a hardcoded user. Every new visitor receives a random dog ID. That ID scopes the profile, check-ins, Snowflake history queries and Solana metadata. Start at /app and BarkPass is about your dog, not mine.

Try BarkPass

🔗 Create a BarkPass for your dog: https://barkpass-dog-days.vercel.app/app

🔗 Explore Bruno's optional sample story: https://barkpass-dog-days.vercel.app/app?sample=1

🔗 Read the source: https://github.com/himanshu748/barkpass-dog-days

The clean app route begins with onboarding. Add a real dog profile, choose a photo, inspect the structured observation, play the voice note, ask a history question and open the passport section.

The Bruno route is only an optional sample story. A history product needs several days before its most interesting screen makes sense, so the sample provides seven check-ins immediately. It is not a separate build or a hidden judging route.

The product flow

The first check-in begins in the browser. BarkPass downsizes an image to a maximum edge of 1,600 pixels, or extracts one representative frame from a short video. This reduces upload time and keeps the provider payload bounded. The prepared JPEG is sent only after the owner chooses the file.

Gemini returns a structured observation. BarkPass saves the normalized result as a check-in, then uses the same result to prepare the dog's voice note. Each additional check-in extends the dog's history. The owner can ask a plain-language question such as "Has Luna's energy changed this week?" and receive an answer calculated from Luna's queried Snowflake rows.

The passport is a separate, optional action. It carries identity and care details, not the owner's private image history. Minting returns public Solana Explorer proof. A shelter tip is prepared as an unsigned devnet transaction and still requires the owner to approve it in Phantom.

The product stays useful even when one provider is unavailable. Local profile and check-in history continue to work. Every non-provider result is labelled in the interface so sample data is never presented as live sponsor output.

The architecture, end to end

The four integrations are stages of one pipeline, not four logos attached to a landing page.

The diagram is a static 1,000 × 1,200 portrait image with large type, so it stays readable at DEV's article width without depending on Mermaid support.

The five layers

Layer	Responsibility	What crosses the boundary
React client	Dog profile state, media preparation, check-in UI, audio playback, Phantom connection and labelled local resilience	A prepared JPEG, normalized observation records, dog identity fields and owner-approved wallet actions
Vercel Functions	Validation, provider orchestration, normalization and credential isolation	Only the minimum provider-specific payload; no provider secret is returned to the browser
Google AI + ElevenLabs	Convert one selected frame into structured visible signals, then convert a bounded sentence into audio	Gemini receives the prepared frame; ElevenLabs receives the derived sentence, never the photo
Snowflake	Durable dog profiles, idempotent check-ins and dog-scoped trend queries	Text identity fields and structured observations; the original profile/check-in photo is not stored there
Solana devnet	Public passport metadata, one-of-one mint proof and unsigned shelter-tip preparation	Selected passport identity fields and the connected public wallet address

One check-in request sequence

Prepare in the browser. A chosen photo is resized to a maximum edge of 1,600 pixels and encoded as JPEG at 0.84 quality. For a short video, BarkPass seeks to an early representative frame and sends that frame rather than the whole clip. Nothing leaves the browser before the owner chooses media.
Observe through POST /api/analyze. The function checks the method, media type and payload size before calling gemini-2.5-flash. Gemini is constrained to a JSON schema. The function parses and normalizes the response again, clamps energy and confidence to their allowed ranges, caps visible flags, and derives the bounded voice sentence on the server.
Speak through POST /api/voice. Only that short sentence is sent to ElevenLabs. The function limits it to 500 characters and returns private-cacheable audio/mpeg; the browser creates a temporary object URL for playback.
Remember through POST /api/dogs and POST /api/checkins. Snowflake receives the current random dog_id, profile fields and the normalized observation. MERGE makes profile and check-in retries idempotent instead of duplicating history.
Answer through POST /api/query. The function synchronizes at most 30 recent local check-ins, selects only rows for the requested dog_id, and computes the row count, average, range, direction and most common mood from the returned values. The prose is assembled from those facts; an unrestricted model does not invent the trend answer.
Carry through POST /api/solana/mint. The server validates the current dog, creates a same-origin metadata URL and uses Metaplex to mint a one-of-one BARK token on devnet. The response returns both the mint and transaction Explorer URLs.
Preserve owner control through POST /api/solana/tip. The server builds a 0.01 SOL devnet transfer with the connected wallet as fee payer, but deliberately does not sign it. The browser hands the serialized transaction to Phantom; only the owner can approve and broadcast it.

Identity and data isolation

The same random dog ID is the join key across browser storage, BARKPASS_DOGS, BARKPASS_CHECKINS, history queries and passport metadata. It is generated when a real profile is created and preserved when that profile is edited. Replacing the optional Bruno sample clears every sample-only field and creates a fresh ID, so a new owner cannot inherit Bruno's photo, microchip, vaccination or seven-day history.

Snowflake has two durable entities:

BARKPASS_DOGS     dog_id → name, breed, age, microchip, vaccination
BARKPASS_CHECKINS (checkin_id, dog_id) → date, mood, energy,
                    posture, flags, confidence, summary

The compound check-in key matters: checkin_id makes retries safe, while dog_id prevents one dog's retry from overwriting another dog's record. Query parameters are bound rather than concatenated, and history reads always include WHERE dog_id = ?.

Trust boundaries and failure behavior

The browser is treated as public. Gemini, ElevenLabs, Snowflake and Solana credentials exist only in Vercel's server environment. Functions validate and bound every incoming field before a provider call. Snowflake uses RSA key-pair authentication through a dedicated least-privilege service role, while the Solana authority is limited to devnet. The shelter-tip path is intentionally split: the server can prepare a transaction, but Phantom retains the owner's signing authority.

Each client adapter also has an explicit resilience boundary. If vision is unavailable, a clearly labelled sample observation keeps the interface demonstrable. If voice fails, device speech can keep the ritual audible. If Snowflake is unavailable, the current browser retains local history and computes the same transparent statistics. On-chain actions never pretend to succeed: live responses return Explorer evidence, verified examples are labelled as examples, and a real tip cannot leave the wallet without Phantom.

This separation is what lets BarkPass degrade one capability without collapsing the whole daily check-in—and without presenting fallback output as live sponsor output.

How Google AI observes without pretending to diagnose

Gemini does not receive a vague "how is this dog feeling?" prompt. The server gives it a narrow role: careful veterinary behavior observer. It may describe only visible signals, must avoid diagnosis and must return JSON matching a strict schema.

const schema = {
  type: 'object',
  additionalProperties: false,
  required: [
    'mood',
    'energy_level',
    'posture_notes',
    'health_flags',
    'confidence',
  ],
  properties: {
    mood: {
      type: 'string',
      enum: ['anxious', 'relaxed', 'playful', 'alert', 'tired'],
    },
    energy_level: { type: 'integer', minimum: 1, maximum: 10 },
    posture_notes: { type: 'string' },
    health_flags: { type: 'array', items: { type: 'string' } },
    confidence: { type: 'number', minimum: 0, maximum: 1 },
  },
}

The endpoint uses Gemini's JSON response mode with temperature 0.2, then normalizes every field again before rendering it. The closed mood vocabulary and numeric bounds make daily observations comparable. A paragraph that changes tone every day would be difficult to chart or query. A stable record is useful.

The most important product rule is: describe the frame, do not invent the dog. A low confidence score stays visible. A visible flag becomes a reason to look again, not a medical conclusion.

I ran five licensed dog photographs through the live provider path. Gemini returned four distinct moods and energy values from 4 to 9.

Photo	Mood	Energy	Confidence	API latency
1	Playful	6/10	0.80	10.53 s
2	Playful	9/10	0.90	4.30 s
3	Anxious	4/10	0.40	8.38 s
4	Relaxed	7/10	0.90	6.81 s
5	Alert	7/10	0.80	4.68 s

Average Gemini latency was 6.94 seconds. The test matters because a vision feature that returns "happy dog" for every photograph is only decoration.

Why ElevenLabs is more than a play button

A score is useful. A sentence is memorable.

BarkPass turns the normalized observation into one short first-person line. High energy produces a more active suggestion. Low energy produces a quieter one. A visible flag asks the owner to take another look without claiming a diagnosis. That bounded line is sent to ElevenLabs using eleven_flash_v2_5 and returned as a 44.1 kHz, 128 kbps MP3.

A timed live call produced an 84,889-byte MP3 in 1.55 seconds. The final Production recheck returned HTTP 200 with a playable 52,288-byte MPEG; output size varies with the generated sentence. Combined with the slowest timed Gemini call, the measured provider path was 12.08 seconds, inside the brief's 15-second target.

If the provider cannot respond, BarkPass can use device speech so the daily ritual does not dead-end. The interface labels that output as a fallback. Device speech is resilience, not evidence of ElevenLabs usage. The provider-backed measurement and playable MP3 are the evidence.

That distinction matters because BarkPass is not claiming to translate dogs. The voice is an interface for recall. Owners are more likely to remember "I am taking things slowly today" than a row containing energy_level: 4.

Why Snowflake is the product's memory

One observation is a moment. Several observations become a pattern.

Profiles are merged into BARKPASS_DOGS. Check-ins are merged into BARKPASS_CHECKINS using both checkin_id and dog_id. The compound match makes retries idempotent and prevents a repeated save from creating duplicate history.

MERGE INTO BARKPASS_CHECKINS AS target
USING (...) AS source
ON target.checkin_id = source.checkin_id
AND target.dog_id = source.dog_id
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED THEN INSERT (...)

History questions query only the requested dog's rows:

SELECT checkin_date, mood, energy_level, health_flags, summary_text
FROM BARKPASS_CHECKINS
WHERE dog_id = ?
ORDER BY checkin_date DESC
LIMIT 30

The returned sentence is calculated from those energy values and mood counts. BarkPass reports the row count, average and range beside the answer. It cannot invent a trend that is absent from the query result.

The deployment uses an X-Small auto-suspending warehouse and a dedicated service user. Key-pair authentication is preferred over a password, and the runtime role is limited to BarkPass's database, schema, tables and warehouse.

In live Production verification, BarkPass stored Maple and three check-ins with energy values 5, 7 and 8. The API reported an average of 6.7, a range of 5 to 8 and a higher final value. A fresh Snowflake worksheet aggregate independently returned the same row count, average, minimum and maximum.

Snowflake is not an analytics screenshot added after the product. It is why BarkPass can answer "Has her energy changed?" with the supporting numbers beside the sentence.

Why Solana carries the passport

A pet passport is more useful when it is not trapped inside one application's database.

The mint endpoint validates the dog first, then generates a public metadata URL containing name, breed, age, microchip ID and vaccination date. Metaplex creates a one-of-one NFT on Solana devnet with symbol BARK, zero royalties and BarkPass's server vault as the mint authority.

Maple's BarkPass was minted from the public Production route as B3FRp9ndjjiVbFwb7MKruabsHicPQQCnL6JdksErEWwa. The mint transaction finalized without error. Explorer shows the BARK non-fungible token, Maple's public metadata URL and BarkPass's new funded devnet vault as authority.

The shelter-tip endpoint does something deliberately different. It prepares a 0.01 SOL devnet transfer with the connected wallet as fee payer, but the server does not sign for the owner. Phantom receives the serialized transaction and must explicitly approve it before broadcast.

Devnet SOL has no monetary value. The proof is the verifiable lifecycle: generate dog-specific metadata, mint once, return Explorer evidence and preserve wallet approval for a transfer.

The hardest engineering decisions

Making the app about every dog, not Bruno

The landing page needed a memorable dog, but the product could not inherit that dog's identity. Profile creation now generates a random dog ID. Check-ins are stored under that ID in the browser and in Snowflake. Passport metadata is created from the current profile. A contract test specifically asks for Luna's history and fails if Bruno appears anywhere in the response.

Keeping a four-provider flow understandable

It would have been easy to show four independent buttons. That would satisfy a checklist but create no product. BarkPass instead gives each provider one job in a single narrative: Gemini notices, ElevenLabs gives the moment a voice, Snowflake remembers and Solana lets the passport travel.

Designing honest failure states

Provider credentials stay in server environments. The public deployment does not expose billable keys to the browser. If a provider is unavailable, BarkPass keeps local history usable and labels the resulting source. It never paints a fallback response as live AI, Snowflake or on-chain activity.

Making a weekend build feel complete

The app supports photo and short-video input, loading, empty, success and error states, keyboard focus, reduced motion, 390-pixel mobile layouts and desktop layouts. The first-run reveal transition was browser-tested after it briefly produced a blank dashboard. The Solana function bundle also needed rpc-websockets pinned to 9.3.8 so Vercel's CommonJS build received a compatible uuid release.

Privacy, safety and reliability boundaries

Profile photos stay in the owner's browser.
A prepared check-in frame is sent only after the owner selects a file.
Text profile details and check-ins sync to Snowflake when that route is available.
Dog IDs scope every profile, query and passport.
Local profiles and history remain usable when a provider is unavailable.
Fallbacks and sample data are explicitly labelled.
Passport minting uses Solana devnet.
Shelter tips require Phantom approval.
BarkPass is a wellness companion, not veterinary advice.

The public client bundle is intentionally secret-free. All four real provider paths now run through the public Production deployment's same-origin server functions, while the repository contains the complete server implementation and reproducible verification record. No provider key, Snowflake private key or Solana vault secret is shipped to the browser.

How I tested it

The Node test suite covers provider contracts and the isolation boundaries that would be easiest to get wrong:

Gemini JSON normalization and the generated voice line
playable ElevenLabs audio bytes
incomplete Snowflake check-in rejection
Snowflake key-pair authentication without password exposure
integration status with key-pair credentials
dog profile validation
dog-specific history with no Bruno inheritance
dynamic Solana metadata for the requested dog
validation before any mint creates chain state

Run the same checks locally:

npm install
npm test
npm run build

All thirteen contract tests pass. The production Vite build passes. I also verified the first-run flow, sample-to-personal profile isolation, the optional seven-day sample, responsive behavior at 390 pixels, public metadata, the finalized devnet mint, a playable ElevenLabs MP3, the five-photo Gemini set and Snowflake's independent aggregate.

Run it yourself

Clone the repository and copy the environment contract:

git clone https://github.com/himanshu748/barkpass-dog-days.git
cd barkpass-dog-days
cp .env.example .env.local
npm install
npm run dev

The interface works locally with labelled resilience paths. To exercise live providers, configure the values documented in .env.example and point VITE_API_BASE_URL at a running functions deployment. The core server variables are:

GEMINI_API_KEY
ELEVENLABS_API_KEY
SNOWFLAKE_ACCOUNT
SNOWFLAKE_USER
SNOWFLAKE_PRIVATE_KEY_BASE64
SNOWFLAKE_WAREHOUSE
SNOWFLAKE_DATABASE
SNOWFLAKE_SCHEMA
SOLANA_RPC
SOLANA_VAULT_KEY

Real values belong only in local or deployment secrets. They must never be committed.

Prize Categories

I am entering BarkPass for Overall Winner, Best Use of Google AI, Best Use of ElevenLabs, Best Use of Snowflake and Best Use of Solana.

The theme connection is direct: the whole product begins with a dog owner's daily photo. The creative choice is to turn that familiar habit into something the owner can hear, question and carry. The technical execution is the connected pipeline and its source boundaries, not the number of APIs used.

Known limitations

BarkPass is a weekend prototype, not a clinical product. Visual observations can be wrong, which is why confidence stays visible and the language remains non-diagnostic. The Production app has all four provider routes configured, while labelled local resilience keeps a check-in usable during a provider outage. The shelter-tip transaction has been prepared successfully, but final broadcast still requires an owner's Phantom approval. There is no account system or multi-device sync yet.

These are explicit boundaries, not hidden promises.

What comes next

The next version would add authenticated multi-device profiles, owner-controlled sharing with carers or veterinarians, reminders, exportable wellness summaries and rate-limited public provider routes. I would also evaluate Gemini against a larger, more diverse photo set and make the passport updateable through an owner-controlled authority.

Most pet apps begin by asking owners to become better record keepers. BarkPass begins with something they already do: take a photo.

That small habit becomes a story they can hear, question and carry with their dog.

I Bought a ₹6 Share and Learned the Hard Way: Building FinEd Saathi in 10 Days

Himanshu Kumar — Sat, 15 Aug 2026 04:47:37 +0000

I once bought a share at about ₹6 without understanding what the complete transaction would cost. I later sold it around the same price and was surprised to see a loss of roughly ₹50 somewhere in my account. I did not know whether that number came from the contract note, ledger, available funds or P&L view. The important lesson was not that every small trade has the same charge. It was that I had acted without understanding brokerage, taxes or where a number in a broker app actually came from.

That experience became the starting point for FinEd Saathi, the voice-first financial literacy tutor I built during 10 Days of Voice Agents - VoiceForBharat Edition. I chose the Financial Services track because I wanted to solve a problem I had felt myself. The challenge listed Financial Services and Learning & Literacy as two separate tracks. I formally stayed in Financial Services, but I deliberately merged finance with education in the product because access without understanding was the problem I wanted to solve. The tutor is for beginners who want Indian market concepts explained in a patient conversation before they put real money at risk. The public code is in the FinEd Saathi repository.

The problem I wanted to solve

Financial education often assumes that a learner already knows the vocabulary. A person who is still asking what an ETF is may be sent to a fee schedule, a tax circular or a dense product page. Even correct information can be hard to use when several unfamiliar concepts arrive at once.

I wanted the product to begin where the learner is. FinEd can explain stocks, mutual funds and SIPs, ETFs, gold, F&O, IPOs and bonds. It can also unpack a confusing charge without inventing a reason. In my ₹6 share story, the agent treats the price profit or loss as zero until I show where the separate loss appeared, such as a contract note, ledger or P&L view. That is more useful than confidently guessing.

The landing page lets a beginner choose a topic before starting a voice lesson.

Why voice changed the product

Voice makes the lesson feel like a conversation instead of an exam. A learner can ask a short question, interrupt or switch between English, Hindi and a code-mixed register. Deepgram Nova-3 handles multilingual speech recognition. Gemini handles the teaching conversation and tools. LiveKit carries the real-time session.

Murf Falcon is the fastest TTS API. I use Murf Falcon 2 with Nikhil as the FinEd voice because his Indian conversational delivery fits a patient tutor. Murf documents Falcon 2 as a real-time speech model with roughly 100 ms time to first audio. The low-latency path matters because a financial explanation quickly feels unnatural when every turn contains a long pause.

The voice workspace shows what the agent heard, which specialist is active and what the learner can do next.

How FinEd Saathi works

The browser sends audio through LiveKit to a Python worker. Deepgram transcribes the speech, Gemini produces a bounded response and Murf Falcon 2 streams the spoken answer. FinEd owns ordinary lessons. TaxEd is a separate specialist reached only after permission. Optional market data, memory, paper trading, human help, outbound telephony and analytics sit outside the essential voice path so missing optional credentials do not break the core tutor.

The main voice session is deliberately compact:

fined_tts = murf.TTS(
    voice="Nikhil",
    style="Conversational",
    model="falcon-2",
    locale="en-IN",  # Initial locale, selected again before each reply stream.
    tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=2),
    text_pacing=True,
)
fined_locale = FinEdTTSLocaleController(fined_tts)

session = AgentSession[SessionState](
    userdata=state,
    stt=fined_stt,
    llm=fined_llm,
    tts=fined_tts,
    turn_detection=inference.TurnDetector(version="v1-mini"),
    vad=ctx.proc.userdata["vad"],
    preemptive_generation=True,
)

Before LiveKit opens each FinEd speech stream, the controller peeks at the actual reply. Devanagari Hindi selects hi-IN; English and Latin-script code-mixed replies select en-IN. That keeps Nikhil fixed while applying a supported locale before Murf snapshots the stream options. TaxEd creates a separate Murf TTS instance using Anusha and a server-normalized en-IN, hi-IN or hi-LATN locale.

What the learner can actually do

A learner can ask for definitions, compare concepts or request a read-only market quote when optional Angel One access is configured. Quote tools can search instruments, return current prices and fetch historical daily closes. They cannot read holdings, positions or account information. If the broker token is missing or expired, the agent says that live data is unavailable instead of guessing a price.

The paper portfolio starts with ₹1,00,000 in virtual cash. It supports simulated NSE EQ cash equity or ETF delivery orders based on a fresh quote. The browser prepares a draft then requires explicit confirmation of that same unexpired draft. The portfolio, virtual cash and fill history stay in browser storage. No real broker order API is called.

Paper trading gives the learner a safe place to practise without connecting a real trading account.

Sessions behave more like a familiar chat product. Meaningful transcripts are stored under local session identifiers in the browser so the learner can switch conversations. Caller memory is separate, private and consent-gated. It saves only learning preferences after a fresh yes. Credentials, government identifiers, account numbers, holdings, income and bank details are excluded.

The analytics page reports anonymous totals such as call count, speaking time, committed handoffs and successful calls. It stores no audio, transcript, utterance text, caller identity or phone number.

Analytics proves whether the experience completed useful actions without turning learners into a surveillance dataset.

Why I built TaxEd as a separate specialist

Investment tax questions need a stricter evidence boundary than a general concept lesson. FinEd first asks whether the learner wants to connect. Only a fresh explicit yes transfers the tax question to TaxEd. Returning to FinEd also requires permission.

TaxEd speaks with Anusha, a Murf Indian voice whose official voice-library entry supports English and Hindi. It searches a packaged registry of official Indian tax rules, states the relevant applicability date and links the source. A rule must be current for the requested date. When a current packaged source is missing, uncertain or past its review date, TaxEd abstains. It does not calculate a person's final liability, file a return or help evade tax.

The handoff is visible, consented and evidence-first.

Safety was part of the architecture

FinEd is education, not investment advice. Deterministic guardrails run before provider inference for real trading, personalized recommendations, guaranteed outcomes, unsafe F&O calls, credential requests and tax evasion. A request to place a real order is refused even when it also mentions paper trading.

The system never asks for a broker password, PIN, OTP, PAN, Aadhaar, bank detail or full account number. F&O mode teaches mechanics, payoff examples and risk only. Human-help requests use a short redacted summary and require fresh consent before storage. Outbound calling is an operator-only optional path that also requires explicit consent. Missing telephony configuration leaves browser voice available and makes the call fail closed.

I documented the deterministic evidence in RED_TEAM.md, including broker identifier redaction during a TaxEd handoff, refusal before model inference and abstention on an unverified tax rule.

The hardest parts

The hardest work was not drawing the interface. It was making state transitions reliable. A handoff that only changes a badge is not a handoff. I had to keep the active specialist, the TTS voice, the transferred question and the return path aligned. I also had to stop repeated permission loops when a clear yes had already been given.

Multilingual speech exposed a similar issue. A multilingual voice identity is not enough when the synthesis locale is fixed. FinEd now routes Nikhil to hi-IN for Devanagari Hindi and en-IN for English before audio begins. TaxEd keeps Anusha while a server-normalized en-IN, hi-IN or hi-LATN locale follows the handoff language. The browser cannot choose arbitrary synthesis settings.

Live market data taught me to design for absence. Broker access tokens expire. The correct fallback is not a stale or invented price. The agent now says the quote is unavailable while concept lessons, the empty paper dashboard and other safe features continue.

I also learned that proof assets are part of product quality. A blurred screenshot can hide the very behavior the article claims to demonstrate. I recaptured the specialist handoff and analytics views so the state, source and privacy boundary are readable.

Run it yourself

You need Python 3.10 through 3.14, uv, Node.js and the repository's pinned pnpm 9 release. Clone the public repository then create local environment files:

cp backend/.env.example backend/.env.local
cp frontend/.env.example frontend/.env.local

Add these required backend values to backend/.env.local:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY

Use the same LiveKit project values in frontend/.env.local and keep the worker name:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
AGENT_NAME=my-agent

Keep real values only in .env.local. Never commit them. Install the backend and its local voice models:

cd backend
uv sync
uv run -m livekit.agents download-files
uv run dotenv -f .env.local run -- python src/agent.py start

In a fresh shell, install and run the frontend:

cd frontend
pnpm install
pnpm dev --port 3001

Open http://127.0.0.1:3001, select a learning mode, choose Talk to FinEd Saathi and allow microphone access and browser audio playback. Ask "What is an ETF?" Then ask how an equity ETF is taxed and say "Yes, connect me to TaxEd" when FinEd requests permission.

Common fixes:

Connection fails: confirm frontend and worker use the same LiveKit project and AGENT_NAME.
Voice is silent: confirm MURF_API_KEY, microphone permission and browser audio playback.
Live quote unavailable: refresh the optional Angel One access token. Do not guess a quote.
TaxEd abstains: verify the packaged official rule is current for the requested date.
Outbound call fails: verify explicit consent, the stored LiveKit SIP trunk and Twilio termination setup.

Angel One, Twilio, the local knowledge index and outbound calling are optional. A normal voice lesson works without them.

How I tested it

The backend has a deterministic suite for guardrails, tools, handoffs, tax rules, memory, analytics and outbound consent. The frontend contracts cover the public interface, browser paper portfolio, token route and documentation. I keep the provider-backed evaluation separate because it requires valid external credentials and inference access.

cd backend
uv run pytest -q --ignore=tests/test_agent.py
uv run ruff check .
uv run ruff format --check .

cd frontend
node --test tests/*.test.mjs
pnpm exec tsc --noEmit
pnpm format:check
pnpm build

The red-team record states exactly which deterministic tests support each claim. I prefer that to describing a manual demo as proof of every failure path.

What I would build next

I would make the source registry easier to update with a review workflow, add more broker-independent historical lessons and test interruption latency across a wider set of Indian network conditions. I would also add an export that lets a learner keep a private summary of concepts they understood without exporting a transcript.

Ten days turned one confusing small trade into a product I personally find smooth to use. More importantly, it changed how I think about a finance assistant. The best version is not the one that sounds most confident. It is the one that teaches clearly, proves its sources, asks permission at boundaries and knows when to stop.

Tracing a multi-agent LLM system: otel-swarm and a SigNoz dashboard pack

Himanshu Kumar — Sun, 26 Jul 2026 17:40:05 +0000

The problem

A single LLM call is easy to reason about. You send a prompt, you get tokens back, you log the latency and you move on.

A swarm is not that. Four or five agents run, some in parallel, each with its own model. One of them times out and quietly falls back to a cheaper model. A critic agent reads the output and sends work back for another round. When the whole thing takes 40 seconds instead of 12, or costs three times what you budgeted, you have no idea which agent did it. Your logs are a flat stream of interleaved lines from concurrent tasks, with no parent-child structure and no way to ask "which role burned the tokens".

Traces solve this exactly. The catch is that hand-wiring OpenTelemetry across every role, every provider call and every retry path is boring work that nobody wants to do twice, and if you skip a level the trace tree lies to you.

So I extracted the instrumentation out of DevSwarm (a multi-agent code generator I built for this hackathon) into a standalone MIT library: otel-swarm.

The API

One createSwarm() call, then three verbs.

import { createSwarm } from 'otel-swarm';

const swarm = createSwarm({ service: 'my-swarm', otlpEndpoint: 'http://localhost:4318' });

await swarm.task('generation', { 'my.prompt': prompt }, async (root) => {
  const plan = await swarm.agent('planner', () =>
    swarm.llm('planner', {
      model: 'primary-model-id',
      fallbackModel: 'fallback-model-id',
      call: async (model) => {
        const res = await yourProviderCall(model);
        return { content: res.text, inputTokens: res.usage.in, outputTokens: res.usage.out };
      }
    })
  );

  await swarm.agent('critic', async (span) => {
    swarm.reviewEvents(span, issues);   // each issue becomes a critic_catch span event
  });
});

swarm.events.on('event', (e) => { /* llm_start, llm_end, fallback, critic_catch */ });

task() opens the root span, agent() opens a child span per role and llm() opens a GenAI-semconv span per model call. call(model) is invoked again with fallbackModel if the primary throws, and the switch is recorded as a fallback_promotion span event carrying from, to and the verbatim provider error.

The last line matters more than it looks. swarm.events is an EventEmitter that mirrors the span lifecycle, and LLM events carry the traceId. A live UI reads the emitter, the tracing backend reads the OTLP exporter and both are fed from the same code path, so the dashboard and the UI can never disagree about what happened. The traceId means a row in your UI deep-links to the exact trace in SigNoz.

Running npm run example against a local SigNoz produced 9 spans under service otel-swarm-demo: a root generation span of 1325ms, then agent.planner/llm.planner, agent.frontend/llm.frontend, agent.backend/llm.backend and agent.critic/llm.critic. llm.frontend carried a fallback_promotion event and agent.critic carried a critic_catch event. The whole story is one trace.

In production use inside DevSwarm the same library has traced 29 generations, 243 model calls across 8 models, 3.78 million tokens and 257 review catches.

Lesson 1: put the fallback in a span event, never on the attribute you group by

This one cost me an afternoon.

An earlier version of the library did the obvious thing when a primary model failed: it overwrote gen_ai.request.model with the fallback's name, so the span would "tell the truth" about which model actually answered.

That is wrong, and it is wrong in a way that is hard to see. Every dashboard panel that groups by model then attributes the primary's failure, its timeout and all of its wasted latency to the fallback that cleaned up after it. My "tokens and latency by model" table showed a cheap fallback model with terrible p95 latency and my expensive primary looking flawless, because every time the primary blew up its cost was silently reassigned to whoever picked up the pieces. I spent an afternoon convinced the wrong model was slow.

The rule that comes out of it: the attribute you group by must record the model you attempted, not the model that ended up answering. The promotion is a discrete thing that happened during the span, and a discrete thing that happened during a span is what span events are for:

span.addEvent('fallback_promotion', { from: model, to: fallbackModel, reason: String(err.message || err) });

Now the grouped panel keeps attributing failure to whoever caused it, and "how often does this role get promoted" is a separate query over events. Two different questions, two different storage locations, no cross-contamination.

The general form: if a value can change mid-span, it does not belong on an attribute you aggregate over.

Lesson 2: reading inside span events in ClickHouse is not obvious

The consequence of putting things in events is that you now have to query events, and SigNoz stores them as an array of JSON strings on the span row. There is no autocomplete that will lead you here. The pattern is ARRAY JOIN to flatten the array into one row per event, then JSONExtractString to reach into the event's attributeMap:

SELECT JSONExtractString(e, 'attributeMap', 'severity') AS severity,
       JSONExtractString(e, 'attributeMap', 'target') AS target,
       count() AS catches
FROM signoz_traces.distributed_signoz_index_v3
ARRAY JOIN events AS e
WHERE serviceName = '{{.service}}' AND name = 'agent.critic'
  AND JSONExtractString(e, 'name') = 'critic_catch'
GROUP BY severity, target ORDER BY catches DESC

That gives you review findings broken down by severity and by what they were aimed at, straight out of span events, with no separate metrics pipeline.

When you only need to know whether an event fired at all, do not pay for the ARRAY JOIN. arrayExists with a substring test over the raw array is enough and it keeps one row per span, which is what you want for a time series:

SELECT toStartOfInterval(timestamp, INTERVAL 15 MINUTE) AS ts,
       attributes_string['swarm.role'] AS role,
       count() AS value
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = '{{.service}}'
  AND arrayExists(e -> e LIKE '%fallback_promotion%', events)
  AND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
GROUP BY ts, role ORDER BY ts

The dashboard pack

The repo ships 3 importable SigNoz dashboards (POST each JSON to /api/v1/dashboards). Every query uses a {{.service}} dashboard variable that defaults to otel-swarm-demo, so you point the pack at your own service by editing one dropdown instead of doing find-and-replace across query strings.

Generation Overview answers "is the swarm healthy right now": generation count, review-gate pass rate, seconds per generation over time and a per-role table of calls, average latency, p95 and errors.
LLM Economics answers "where did the tokens go": total tokens, tokens over time by role, a tokens-and-latency table by model and fallback promotions by role (the arrayExists query above).
Review-Gate Funnel answers "is the reviewer doing its job": total catches, regeneration rounds, catches per generation over time and a table of recent generations with verdict, catches and regenerations.

The alert rules

Two rules ship in the v2alpha1 schema (POST to /api/v2/rules).

The first is a fallback-promotion spike. If promotions jump, a provider is degrading and you are silently paying a different bill than you planned.

The second is the one I care about more: a review catch-rate flatline. If your critic agent suddenly stops finding anything, the tempting read is that the generators got better. In practice the reviewer broke: a prompt change made it answer in a shape the parser drops, or its model started returning empty content and the failure is being swallowed as "no issues found". A quality gate that passes everything is indistinguishable from no quality gate, and it fails silently by construction. Alert on the absence.

The repo also includes casting.yaml and casting.yaml.lock, the Foundry config the SigNoz instance behind these dashboards was installed from, so the backend the pack targets is reproducible rather than assumed.

Install and run

npm install github:himanshu748/otel-swarm

To see the example trace end to end against a local SigNoz:

git clone https://github.com/himanshu748/otel-swarm && cd otel-swarm
npm install
npm run example                                                     # spans to console
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npm run example   # spans to SigNoz

With no endpoint set it exports to the console, so you can check the span tree before wiring a backend.

Closing

The library is about 130 lines. Most of the value is not code, it is the two decisions above: keep mutable facts in span events, and treat "the reviewer found nothing" as a symptom rather than a result. Both were learned by getting them wrong first.

I built otel-swarm with Claude Code, including the dashboard JSON and the ClickHouse queries in this post.

Repo: github.com/himanshu748/otel-swarm, MIT.

We instrumented an AI agent swarm with SigNoz, and its own telemetry told us we were wrong about almost everything

Himanshu Kumar — Sat, 25 Jul 2026 15:01:20 +0000

Built for the WeMakeDevs Agents of SigNoz hackathon, July 2026.

Mission Control. The graph is the swarm, the river underneath it is the live span stream, and every bar deep-links into that trace in SigNoz.

DevSwarm turns one prompt into a working full-stack app. Five open-weight models plan it, build it, review it and repair their own routing. Nothing it produces is trusted blindly, and every step is an OpenTelemetry span in SigNoz, including the steps that go wrong.

We built the observability first, expecting it to prove the thing worked.

It did something more useful. It spent a week proving that almost everything we believed about our own system was wrong. We blamed a model for a limit we had set ourselves. We blamed a provider outage on the model. We assumed our review agent was our strongest link when it was measurably the weakest. And we spent days writing a design system that, when we finally measured it, was making the output worse.

Not one of those was found by reading the code again. Every single one came off a span event, a dashboard row or a benchmark that disagreed with us.

So this is not an architecture post. It is six times the telemetry told us we were wrong, with the queries.

The current numbers, all read live out of SigNoz rather than typed into a slide: 22 generations, 189 traced model calls across 8 models, 2.85 million tokens, 225 critic catches, 19 fallback promotions and 24 generated apps each reporting under their own service name.

What the swarm actually is

Five roles, each on the open-weight model that measured best for that job:

role	model	job
planner	GLM-5.2	turn a prompt into a typed build plan and a locked API contract
frontend	GLM-5.2	one self-contained index.html against that contract
backend	Qwen3-Coder-480B	one Express server against the same contract
critic	Kimi-K2.7-Code	review both, gate the merge, route catches back to their owner
doctor	GLM-5.2	read the swarm's own traces and repair its model routing

Every number on our own landing page is a live ClickHouse query against the trace store. Marketing copy that drifts from the telemetry is impossible by construction.

Everything is served through Hugging Face Inference Providers. There are zero closed-model API calls in the system, which turned out to matter for reasons we did not anticipate (see the provider section below).

The critic is the load-bearing part. Frontend and backend are generated in parallel from the same contract, then an independent model reviews both for contract conformance, security and runtime bugs. Real catches route back to the agent that owns them, that agent patches its own file and the critic re-reviews only the delta. Two regeneration rounds, then it ships with an honest verdict either way.

The same gate runs on changes. Asking a finished app for "a star rating on each book, settable when adding one" re-plans against the stored contract, re-runs only the agents that instruction touches, and puts the result through the identical review. A refinement is its own root span, so a change request is as traceable after the fact as the build that preceded it.

Why we instrumented before we polished

A multi-agent system fails in ways a single-model tool does not. A call can succeed while producing an unusable artifact. A fallback can rescue a request so smoothly that nobody notices the primary is dead. Latency can triple because one role silently started thinking twice as long. None of that shows up in a request log.

So the very first thing that worked in this project was not code generation. It was a trace.

SigNoz is self-hosted through Foundry, which is a single-config install. Our casting.yaml and casting.yaml.lock are committed to the repo so the deployment is reproducible by anyone, judges included.

Three signals, and what each one carries

Traces. Every model call is a span named llm.<role> carrying GenAI semantic conventions:

span.setAttributes({
  'gen_ai.operation.name': 'chat',
  'gen_ai.request.model': model,
  'gen_ai.usage.input_tokens': usage.prompt_tokens,
  'gen_ai.usage.output_tokens': usage.completion_tokens,
  'devswarm.role': role
});

Two span events do the heavy diagnostic lifting. fallback_promotion records that a primary failed, which model took over and the verbatim reason. critic_catch records every issue the review agent found, with its target and severity. Both are events rather than separate spans on purpose: they belong to the call they describe, and they survive in the trace even when the call ultimately succeeds.

Metrics. Six counters and a histogram, because some questions are time series questions rather than trace questions: devswarm.tokens, devswarm.llm.calls, devswarm.llm.duration, devswarm.fallback.promotions, devswarm.critic.catches, devswarm.generations, devswarm.refinements. Labelled by role, model and outcome.

Logs. Structured records for the things a human reads during an incident: a fallback promoting, a doctor diagnosis, a generation completing with its verdict and catch count. Same resource attributes as the traces, so a log line and a span line up.

The whole trace layer is now extracted into a small library, otel-swarm, that any multi-agent system can drop in. DevSwarm consumes it as a real dependency, which means if the library breaks, our own dashboards go dark first. That felt like the honest way to ship it.

One generation as a flame graph. Planner, then frontend and backend in parallel, then the critic.

The log stream during a rough run. WARN lines are fallback promotions, each naming the model that failed and why.

Reading the swarm out of ClickHouse

Five dashboards, all committed as JSON in observability/dashboards/. The one we actually live in is Command Center: is the swarm healthy, and if not, which role. The odd one out is Born Observable, which contains no swarm data at all, only the generated apps reporting under their own service names.

Three queries worth sharing, because span events in ClickHouse are not obvious the first time.

Role health, straight off the trace store:

SELECT attributes_string['devswarm.role'] AS role,
       count() AS calls,
       countIf(statusCode = 2) AS errors,
       round(quantile(0.95)(durationNano) / 1e9, 1) AS p95_s,
       sum(attributes_number['gen_ai.usage.input_tokens']
         + attributes_number['gen_ai.usage.output_tokens']) AS tokens
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'devswarm' AND name LIKE 'llm.%'
GROUP BY role ORDER BY calls DESC

Fallback promotions over time, which requires reaching into the events array:

SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS ts,
       attributes_string['devswarm.role'] AS role,
       count() AS value
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'devswarm'
  AND arrayExists(e -> e LIKE '%fallback_promotion%', events)
GROUP BY ts, role ORDER BY ts

The one that took longest to work out reads inside the events, so you can ask what the review gate actually caught rather than how many times it caught something:

SELECT JSONExtractString(e, 'attributeMap', 'severity') AS severity,
       JSONExtractString(e, 'attributeMap', 'target') AS target,
       count() AS catches
FROM signoz_traces.distributed_signoz_index_v3
ARRAY JOIN events AS e
WHERE serviceName = 'devswarm' AND name = 'agent.critic'
  AND JSONExtractString(e, 'name') = 'critic_catch'
GROUP BY severity, target ORDER BY catches DESC

The answer, over the week: 221 high, 46 medium, 10 low, and the frontend agent is on the receiving end of 70 percent of them. That single row changed how we think about the swarm. The half of the system generating markup and client-side state is where the bugs live, not the half touching the database.

One design decision we are glad about: the marketing numbers on our own landing page are fetched from these same queries at request time. The page cannot drift from the telemetry, because there is only one source of both.

Command Center. Top row answers "is the swarm healthy", the role-health table answers "which role", while the fallback chart should trend to zero.

LLM Economics: tokens and latency per role and per model, which is how we caught the frontend role burning two thirds of its budget on reasoning.

What one app actually costs

Every token in the table below came off a span. This is one real run, the letterpress site above, priced at the rates the Hugging Face router itself reports for the providers we use.

role	model	in	out	cost
frontend	GLM-5.2	27,855	38,888	$0.2101
critic	Kimi-K2.7-Code	50,996	14,539	$0.1066
planner	GLM-5.2	670	4,090	$0.0189
backend	Qwen3-Coder-480B	2,800	3,778	$0.0069
		82,321	61,295	$0.34

Thirty four cents for a designed marketing site with a working Express backend, a waitlist that validates email and rejects duplicates, plus its own OpenTelemetry wiring. Across passing runs the range is about 18 cents to 56 cents.

Two things in that table surprised us.

The critic costs fifteen times what the backend author costs. Reviewing the code is dramatically more expensive than writing it, because review means reading both artifacts in full, twice, while the backend agent writes one file once. Nobody budgets for that. If you are building a review gate into an agent system, it is not a rounding error on top of generation, it is a third of your bill.

And a failed run costs more than a successful one. Our worst generations burned 232,000 tokens hitting the regeneration ceiling, against 74,000 for the cleanest pass. So convergence is not only a quality metric, it is the cost metric. Fixing the contract-format bug in finding five did more for our unit economics than any model swap we made.

Alerts that wake an agent instead of a human

This is the part of the build we are proudest of, and it is a genuinely small amount of code.

Two alert rules live in observability/alerts/: a fallback-usage spike and a critic catch-rate flatline. Both notify a webhook channel called swarm-doctor, which points at POST /api/doctor/webhook on the swarm itself.

When an alert fires, the Doctor wakes up, queries the swarm's own traces for the last hour and decides what to do about the routing table. It promotes a backup model, resets a recovered primary or does nothing, then explains itself in plain English using the numbers it just read. Its first real diagnosis, verbatim:

"The critic role is the clear problem area: its primary triggered 6 fallback promotions out of 9 calls (67%) with a 22% error rate and p95 latency of 242s. Backend, planner and doctor are healthy."

The Doctor's own model calls are traced too, so the healer is exactly as observable as the patient.

The Doctor reporting a healthy swarm. It read 180 minutes of its own traces to say so, and it is honest about sample size: "call volume is very low, so latency figures are not statistically meaningful".

Two hard-won SigNoz API notes, since we lost hours to both:

Alert rules must be created against /api/v2/rules with schemaVersion: v2alpha1, a notificationSettings block and at least one channel. The v1 endpoint accepts the request and returns "alert rule is not valid" with no indication of which field is wrong. Dashboards, by contrast, go to /api/v1/dashboards with a SIGNOZ-API-KEY header and behave exactly as documented.

Also: a cold Docker restart can leave ClickHouse replicas read-only until Keeper reconnects. It usually self-heals within a minute. If it does not, SYSTEM RESTORE REPLICA per table clears it.

Apps that are born observable

Every app the swarm generates ships instrumented. Alongside index.html and server.js, each generated folder gets an otel.mjs bootstrap, a package.json, and a signoz-dashboard.json scoped to that app's own service name. If a SIGNOZ_API_TOKEN is configured, the dashboard is created in SigNoz at generation time, before the user has opened the preview.

So the generated app appears in SigNoz as its own service, with RED metrics and a routes table, seconds after it exists. Twenty four of them are in our instance right now.

Worth saying because people assume otherwise: there is no image model anywhere in this pipeline. The swarm generates 227 inline SVG elements across the 26 apps it has built, an average of 8.7 per app, and every one of them was written as markup by a language model. The Vandercook press in the screenshot above is hand-drawn SVG, not a generated image. The only assets we ever image-generated are DevSwarm's own favicon and social card, which are branding for the tool rather than anything the swarm produces.

One of the outputs. The Vandercook press is inline SVG the model drew itself, and this app reports to SigNoz under its own service name from the moment it boots.

One caveat we had to learn the hard way: an app only emits spans while its backend is actually running. Early on our preview served the frontend statically, so the generated Express server never booted and the app silently fell back to localStorage. The preview looked perfect and the service page was almost empty. That mismatch, two spans where there should have been dozens, is what gave the bug away. Previews now spawn the real server as a child process and proxy to it.

Six things our own telemetry told us we had wrong

This is the section I would want to read, so it is the longest one.

1. A "model limitation" was a stale constant we wrote ourselves.

Traces showed every frontend failure as finish: length, truncating full-page HTML. We concluded GLM-5.2's provider capped completions at 16384 tokens and moved the role to another model. The cap was real when we found it. It was also in our own config, and when the provider limit later lifted, our constant kept enforcing a limit that no longer existed. The average frontend artifact needs about 19,000 output tokens. We had guaranteed truncation and blamed the model for a fortnight.

2. The real cause was provider roulette, visible only in the span event text.

After removing our own cap, GLM still failed intermittently with a 400: max_completion_tokens is limited to 16384 for glm-5.2. The Hugging Face router load-balances a model across every provider serving it, and their limits disagree. We probed all seven: scaleway caps at 16384, featherless at 32768, novita and zai-org at 131072, while together, fireworks-ai and deepinfra accept 200000 or more. Unpinned, roughly one request in seven hit the strict provider and died instantly. Pinning the model to one provider produced our first ever generation with zero fallbacks. That entire diagnosis came out of the reason attribute on a fallback_promotion event.

3. Our span attribute was hiding the failures we most wanted to see.

When a primary failed, our code overwrote gen_ai.request.model on the span with the fallback's name. It seemed tidy. It meant every dashboard row attributed the primary's failure, and its wasted latency, to the fallback that cleaned up after it. We spent an afternoon convinced the critic's backup model was slow and error-prone. Isolating them in a benchmark showed the opposite: the backup was fine at 7 seconds, and the primary was the problem. If you take one implementation detail from this post, take this one. Record the model you attempted, and put the promotion in an event, not on top of the attribute you will later group by.

4. Our review gate was the weakest model in the swarm.

We had never benchmarked the critic, so we built one: a generated app with three documented contract defects, three runs per model, scored on defect recall. DeepSeek-V4-Pro, our incumbent primary, found 2 of 9. One run burned its entire 32768-token budget and returned nothing parseable. Kimi-K2.7-Code found 8 of 9 and was consistent across runs. The clearest pattern in the data was that on a review task, reasoning volume tracks defect recall: the terse models answered in under 250 output tokens and missed real bugs.

5. A 20 percent pass rate was one missing sentence in the contract.

Our plans specified field names and types but never formats, ranges or nullability. So the backend rejected rating: 0 while the frontend sent 0 as its default, and the backend demanded YYYY-MM-DD while the frontend sent full ISO strings. Three consecutive generations hit the regeneration ceiling on exactly this class of disagreement. The fix was making the planner write a binding rules string per field, for example "integer 0 to 5 inclusive, where 0 means unrated and is a valid value". Both builders now read the same sentence. Catches dropped from 9 to 3 and the next run passed.

6. Our design system was making the output worse.

We wrote a careful design guide so generated apps would not look like generated apps. Then we measured it: same model, same prompt, the only variable being whether the guide was attached. Without it, 14 inline SVGs, 3 animations and a deliberate typeface pairing. With it, 5 SVGs, 1 animation and Courier New. Three of our own rules did that. "System font stack is fine" told the model not to bother choosing type. "Cut any animation that does not serve the subject" read as licence to strip ornament. And our frontend prompt banned all external requests, which silently banned Google Fonts, so it could not have chosen a real typeface even if it wanted to. We had written a list of prohibitions, which is good at preventing bad output and bad at producing good output.

Before. Mono labels from a system stack, and card colours the backend invented at random.

After. Same model, same prompt, three rules removed from our guide: Fraunces display type, a drawn logo mark, filter chips carrying live counts, plus the books rendered as spines on a shelf.

There was a second layer to that one. The guide had a generous section for marketing sites, licensing scroll reveals, entrance sequences and layered depth, plus a stingy section for apps. Every app the swarm built was being held to a deliberately plainer standard than every site, and nobody had noticed because the sites looked great.

How this was built

DevSwarm was built with Claude Code, which is worth stating plainly rather than leaving as an inference. An AI coding agent helped build an AI coding agent, and the hackathon rules ask entrants to declare assistant use, so here it is.

It is also relevant to the point of this post. Every finding in the section above started as a confident, wrong belief held by both of us, human and assistant alike. The stale token cap, the model we blamed for a provider's limit, the review gate we assumed was our strongest link, the design system we were sure was helping. None of those were resolved by reasoning harder about the code. They were resolved by a span event, a dashboard row or a benchmark disagreeing with us.

That is the argument for instrumenting an agent system early. When you are building with agents, and with an agent, the telemetry is the only participant in the conversation with no opinion to defend.

What is worth copying from this build

If you are instrumenting an agent system, three things paid for themselves immediately.

Put the reason text in the span event. Not a code, not an enum, the actual provider error string. Both of our worst bugs were solved by reading that field, and neither would have been visible in a metric.

Never overwrite an attribute you intend to group by. Add, do not replace.

Make your product read its own telemetry. Our landing page statistics, our Doctor's diagnosis and our dashboards all run the same queries against the same trace store. It removes a whole category of drift, and it turns your observability stack from a debugging tool into a feature.

The uncomfortable part

The through line of every finding above is the same, and I have thought about it more than I expected to.

Not one of these was a hard problem. A stale constant. A provider with a different limit. An attribute overwritten in the wrong place. A missing sentence in a contract. Three over-cautious lines in a style guide. Any of them would have been a five minute fix if we had known. Together they cost us most of a week and made the system look, from the outside, like the models were letting us down.

They were not. Every single time, the model did exactly what our configuration told it to do. The failure was always upstream of the model, in something we had written and then stopped looking at.

I think that is the actual lesson of building with agents, and it is not a comfortable one. The debugging skill is not prompt engineering. It is being willing to believe your instrumentation over your own memory of what you configured three days ago. We only got there because the telemetry kept producing numbers that made our explanations impossible.

If you are building something similar, I would genuinely like to know whether your experience matches. My suspicion is that a lot of "the model is not good enough" is actually "my config is stale and I have no way to see it".

The bug that never crashed: how I fuzzed an AI's own code sandbox and found it lying to its model

Himanshu Kumar — Wed, 22 Jul 2026 13:26:43 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The bug that never crashed

The scariest bug I caught this month never threw a stack trace. It never paged anyone. It just quietly made an AI agent dumber, on repeat, and handed the blame to the model.

Here is the whole story, because the way I found it turned out to matter more than any single fix.

Where it started

smolagents is Hugging Face's agent framework, 28k+ stars, and it has an unusual design: the agent writes Python as its reasoning, and a sandboxed interpreter (LocalPythonExecutor) runs that code. The model thinks by coding. So the sandbox is not a side feature. It is the surface the model lives on.

I went in looking for one bug to fix for this challenge. Every obvious open issue was already claimed or had a competing PR. So instead of reading the issue tracker, I did something almost stupid: I fed the sandbox ordinary, boring, valid Python and watched what it refused to run.

The rule I gave myself: if it is valid Python and the sandbox rejects it, that is a bug. The model writes valid Python. If the sandbox chokes on it, the agent pays.

The first false rejection

config = {**{"temperature": 0.7}, "top_p": 0.9}

Merging dicts with **. Every LLM writes this. The sandbox's answer:

InterpreterError: NoneType is not supported.

There is no None in that line. I stared at it, then went to the source. In Python's AST, a **spread entry inside a dict literal has None where the key would be, a marker that says "this is a spread." smolagents walked the keys and tried to evaluate that None as if it were an expression. So the spread marker got blamed on a value the developer never wrote.

That is when it clicked. This was not a crash bug. It was a lying bug.

Why lying is worse than crashing

Here is the part that turned a one-off fix into an obsession.

The error was handled. smolagents catches it and feeds it back to the model as guidance: "here is what went wrong, try again." Good design, normally. But the guidance was wrong. The message said NoneType, and the model's code had no None, so the model could not act on it. It did the only thing a faithful agent can do with a message that is already a lie: it retried the exact same valid code.

And again. And again.

I only saw the loop because I had wired the demo to Sentry. One issue. Three events. Three identical failures on one line, each burning a real LLM call and a slot in the step budget, until the run gave up and produced a worse answer than it should have. Without something counting the events, you would never see the loop. You would just see an agent that "isn't very good," and you would go blame the model.

A red stack trace is mercy. It tells you where to look. The polite, well-handled, misleading error is the one that eats your afternoon, or in this case, eats the agent's entire budget while looking like helpful feedback.

Sentry's Seer read the same event and reached the same root cause I did, independently: None keys in ast.Dict fed to the evaluator, agent retries in a loop. That was the moment I trusted the pattern enough to go hunting.

The pattern held everywhere I looked

Same fuzzer, same rule, more boring valid Python. The sandbox kept lying:

10 ** 10 ** 8 did not error at all. It froze the entire process, forever, and the execution timeout never fired, because a single big-integer operation runs as one uninterruptible C call that holds the GIL and never lets the timeout thread wake up. The faulthandler dump showed the main thread stuck before the timer even armed.
best, *rest = scores failed with "Cannot unpack tuple of wrong size." There was no wrong size. The sandbox simply never implemented starred unpacking, a feature Python shipped in 2008.
a, b = "hi" was rejected outright. Strings unpack fine in real Python.
[a, b] = [1, 2] silently assigned nothing. No error, no values. The quietest failure of all.

Four different corners of the language. One personality: confidently wrong, politely delivered, and invisible unless you were counting.

The fixes, briefly

Each one came down to matching CPython instead of guessing:

Dict spread: evaluate pairwise, a None key means merge a mapping. And gate it on hasattr(value, "keys"), not the Mapping ABC, so duck-typed mappings work too.
The freeze: you cannot interrupt a GIL-holding C call, so refuse to start it. Estimate the result's bit length from the operands in O(1) and raise a real error above a cap, pointing the model at pow(base, exp, mod).
Unpacking: accept any iterable, support one starred target that absorbs the surplus into a list, handle list-pattern targets, and use CPython's exact error messages so that when the model does pass the wrong count, it gets an actionable message instead of a dead end.

Four fixes, four PRs, sixty-plus new tests, all verified failing on main first.

The twist I did not expect

I opened the PRs. Minutes later, OpenAI's Codex reviewer commented on two of them, and it was right both times. On the big-int fix it caught that I had checked type(x) is int, which lets bool and int subclasses slip through the guard. On the dict fix it caught that my Mapping ABC check was stricter than CPython.

So the final scoreboard was: an AI wrote the buggy code, a different AI reviewed my AI-assisted fix and found the hole, and a third AI (Seer) had already confirmed the root cause. I was the one steering, but I spent a good chunk of this project as the human in a loop of machines checking each other. Fitting, for a bug about an agent that could not tell it was stuck.

What I actually learned

Fuzz the boundary with valid input. Everyone fuzzes with garbage to find crashes. The higher-value bugs in an AI system are the false rejections: valid input the system refuses, because that is what silently degrades a model that is doing everything right.
A handled error is not a safe error. If the message is wrong, "handled" just means the failure is quiet instead of loud. Quiet is worse.
Instrument for absence and repetition. The tell for this whole bug class was not an exception. It was the same event, three times. An agent getting a different error each step is exploring. An agent getting the same error from the same input is stuck, and the stuckness is invisible to the agent because the message looks like feedback. Count your events.

The bugs are fixed. The lesson I am keeping is simpler than any of them: the dangerous failures in an agent stack are not the loud crashes. They are the polite, confident, wrong messages that let the model fail on repeat while everyone blames the model.

The evidence

Each one has its own full write-up with Sentry before/after evidence, if you want the deep dives. This was the story of how they all turned out to be the same bug wearing four different masks.

The smolagents sandbox broke 'a, *b = list', one of Python's most common lines

Himanshu Kumar — Tue, 21 Jul 2026 05:46:04 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Fourth entry, and the fuzzer that found entry 3 is still paying out. Same failure family, different corner of the language: valid Python the sandbox refuses to run, with an error that lies about why.

The one-liner every developer writes

best, *rest = scores

Splitting a list into "the first one" and "the rest" is about as ordinary as Python gets. It is PEP 3132, shipped in 2008. Under smolagents' sandbox it fails with:

InterpreterError: Cannot unpack tuple of wrong size

There is no wrong size. scores has four items and the pattern accepts any length of two or more. The message describes a problem that does not exist, so the agent does the only thing a faithful agent can do with a message that is already a lie: it retries the identical, valid code.

And it is not just the starred form. All of these are standard Python and all of them were broken:

first, *rest = "hello"   # Cannot unpack tuple of wrong size
a, b = "hi"              # Cannot unpack non-tuple value
[a, b] = [1, 2]          # silently assigns nothing at all

Why it happens

smolagents runs model-generated code in its own AST interpreter, and assignment targets go through one function, set_value. It handled exactly one shape: a fixed-size ast.Tuple.

elif isinstance(target, ast.Tuple):
    if not isinstance(value, tuple):
        if hasattr(value, "__iter__") and not isinstance(value, (str, bytes)):
            value = tuple(value)
        else:
            raise InterpreterError("Cannot unpack non-tuple value")
    if len(target.elts) != len(value):
        raise InterpreterError("Cannot unpack tuple of wrong size")
    ...

Three separate holes hide in those few lines:

Starred targets. a, *b has two target elements but the value has three items, so len(target.elts) != len(value) fires. The * marker is never even looked at.
Strings and bytes. They are explicitly excluded from the iterable path, so a, b = "hi" is rejected even though CPython unpacks strings happily.
List-pattern targets. [a, b] = ... is an ast.List, not an ast.Tuple, so it matches no branch, falls through, and silently assigns nothing. No error, no values, the worst kind of quiet.

The Sentry angle: count the retries

Same lesson as the last two entries, and Sentry keeps making it visible. The error is handled: the agent catches it and feeds it back to the model as guidance. But the guidance is wrong, so the model cannot act on it, so it loops. One bug, one misleading message, three identical failures burning three steps:

Three events on one issue is the retry loop made countable. Without it you would see a slow run, not a stuck one. Sentry's Seer read the same event and landed on the exact cause:

smolagents' custom Python interpreter does not support starred unpacking (e.g. best, *rest = scores), treating it as a fixed-size tuple unpack. set_value checks len(target.elts) != len(value) and raises, without handling ast.Starred targets.

The fix

Rewrite the branch to match CPython instead of guessing:

elif isinstance(target, (ast.Tuple, ast.List)):
    elts = target.elts
    starred = [i for i, e in enumerate(elts) if isinstance(e, ast.Starred)]
    if len(starred) > 1:
        raise InterpreterError("multiple starred expressions in assignment")
    if not hasattr(value, "__iter__"):
        raise InterpreterError(f"cannot unpack non-iterable {type(value).__name__} object")
    values = list(value)
    if starred:
        i = starred[0]
        n_after = len(elts) - i - 1
        if len(values) < i + n_after:
            raise InterpreterError(
                f"not enough values to unpack (expected at least {i + n_after}, got {len(values)})"
            )
        split = len(values) - n_after
        # assign head, then the starred target gets the middle as a list, then the tail
        ...
    else:
        if len(values) < len(elts):
            raise InterpreterError(f"not enough values to unpack (expected {len(elts)}, got {len(values)})")
        if len(values) > len(elts):
            raise InterpreterError(f"too many values to unpack (expected {len(elts)})")
        ...

Any iterable now unpacks, a single starred target absorbs the surplus into a list in any position (a, *b, *a, b, a, *b, c), list-pattern targets work, and the size errors read exactly like CPython's, so when the model genuinely does pass the wrong number of values it gets an actionable message instead of a dead end.

After

app: step 1 ok, output = (90, [82, 71, 65])

One step. No loop. best is 90, rest is the tail, the way the model expected all along.

Numbers

4 valid unpacking forms fixed: starred targets, string unpacking, list-pattern targets, and the CPython error messages
Reproduced on current main and 1.26.0; 3 wasted agent steps per occurrence, visible only because Sentry counts events
14 new tests plus one existing test updated to the improved message
411 passing, ruff clean

Links

The through-line across all four entries has not changed: the dangerous agent bugs are not the loud crashes, they are the polite, well-handled, wrong messages that let the model fail on repeat. Fuzz the sandbox with ordinary valid code, and count your events.

The smolagents bug that made my agent retry the same valid code three times

Himanshu Kumar — Mon, 20 Jul 2026 07:19:13 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Third entry in the DEV x Sentry Bug Smash. Entry 1 was a crash with a confusing message. Entry 2 was a freeze the timeout could not catch. This one is quieter and sneakier: valid Python that the sandbox rejects with an error pointing at the wrong thing entirely.

When the open issues run out, fuzz

By entry 3 every obvious open smolagents bug was already claimed or had a competing PR. So instead of reading the issue tracker I pointed a small fuzzer at the piece of smolagents that runs the most untrusted code: LocalPythonExecutor, the sandbox that executes model-generated Python.

The method is boring and effective: feed it ordinary, valid Python one snippet at a time, and flag anything that raises InterpreterError. Valid Python that the sandbox refuses to run is, by definition, a bug, because the model writes valid Python and expects it to work.

That surfaced four unreported bugs in one afternoon. This post is about the one I shipped: dict unpacking.

The bug

config = {**{"temperature": 0.7, "max_tokens": 512}, "top_p": 0.9}

Merging dicts with ** is one of the most common things an LLM writes. Under smolagents it fails with:

InterpreterError: NoneType is not supported.

There is no None anywhere in that line. The message sends you looking for a null value that does not exist.

Why it happens

In Python's AST, a dict literal keeps its keys and values in two parallel lists. For a normal entry the key is an AST node. For a **mapping spread entry, the key is literally None, a signal that says "this is a spread, not a key/value pair."

smolagents evaluated every key by walking expression.keys and calling evaluate_ast(key, ...) on each one. When the key is None, that call falls through every isinstance branch to the catch-all raise InterpreterError(f"{type} is not supported"). So the spread marker got evaluated as if it were an expression, and the model got blamed for a None it never wrote.

Why silence is the expensive part

Here is the part the Sentry view made obvious. The error is handled: the agent catches it and feeds it back to the model as "here is what went wrong, try again." But the message names NoneType, and the model's code has no None, so the model cannot act on it. It retries the exact same valid syntax. And again. Every step burns a real LLM call and a slot in the step budget until the run gives up.

One bug, one misleading message, three identical failures:

Three events on a single issue is not noise. It is the agent stuck in a loop, and without Sentry counting the events you would never see the loop, only a run that quietly underperformed. Sentry's Seer read the same event and reached the exact root cause:

smolagents' LocalPythonExecutor doesn't handle dict unpacking (**) syntax: None keys in ast.Dict cause an unsupported type error. [...] evaluate_ast(None, ...) matches no isinstance branch and falls to the else clause. The interpreter raises InterpreterError: NoneType is not supported, the agent retries with identical code, burning steps in a loop.

The fix

Evaluate the dict pairwise instead of evaluating keys blindly. A None key means "merge this mapping":

result = {}
for key_node, value_node in zip(expression.keys, expression.values):
    if key_node is None:
        value = evaluate_ast(value_node, *common_params)
        if not hasattr(value, "keys"):
            raise InterpreterError(f"'{type(value).__name__}' object is not a mapping")
        result.update(value)
    else:
        key = evaluate_ast(key_node, *common_params)
        result[key] = evaluate_ast(value_node, *common_params)
return result

This matches CPython exactly: spreads merge in order, later keys win, and unpacking a non-mapping raises 'list' object is not a mapping.

A reviewer caught my fix being too strict

I first gated the spread on isinstance(value, Mapping). Minutes after the PR opened, OpenAI's Codex reviewer flagged it (P2): CPython does not require the Mapping ABC, it only requires an object with a keys() method. Since the sandbox lets users define their own classes, a duck-typed mapping with keys() and __getitem__() would have been wrongly rejected. I switched the check to hasattr(value, "keys") and added a test for exactly that case. AI wrote the code, AI reviewed the code, I kept score.

After

On the patched build the same line just runs:

app: step 1 ok, config = {'temperature': 0.7, 'max_tokens': 512, 'top_p': 0.9}

One step, no loop, no phantom None.

Numbers

4 unreported bugs found by fuzzing valid Python through the sandbox; this is the first fix
Misleading NoneType error reproduced on current main and 1.26.0
3 wasted agent steps per occurrence, visible only because Sentry counts events
9 new tests: spreads, double spreads, override order both ways, a duck-typed mapping class, empty spread, non-mapping rejection
406 passing, ruff clean

Links

The pattern across all three entries: the worst agent bugs do not throw a red stack trace at you. They hand the model a plausible-but-wrong message and let it fail politely, on repeat. Count your events.

One line of math froze my AI agent forever. The timeout watched and did nothing.

Himanshu Kumar — Sun, 19 Jul 2026 07:54:58 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

This is my second entry for the DEV x Sentry Bug Smash challenge. Entry #1 was a crash with a confusing error message. This one is the opposite and it is scarier: no crash, no message, no event. Just silence.

The bug that sends you nothing

smolagents runs LLM-generated Python in a sandboxed executor with a timeout. Issue #2473 claims that one line of model-generated math defeats it completely:

from smolagents.local_python_executor import LocalPythonExecutor

executor = LocalPythonExecutor(additional_authorized_imports=[], timeout_seconds=2)
executor.send_tools({})
executor("10 ** 10 ** 8")  # 2 second timeout. Should be fine, right?

I ran this with a 2 second timeout and a faulthandler bomb set for 20 seconds. The timeout never fired. The process sat frozen until the external kill. An agent that generates this expression (and "compute this huge number" is exactly the kind of thing agents try) freezes its host process forever.

Zero comments on the issue, zero PRs. Mine now.

Why the timeout lies to you

smolagents' timeout is thread based: a worker thread runs the code, the main thread waits in future.result(timeout=2). That design is fine for almost everything, because CPython switches threads between bytecode instructions.

But 10 ** 10 ** 8 is not "almost everything". CPython computes arbitrary precision **, << and * inside a single C call that holds the GIL from start to finish. No bytecode boundary, no thread switch, no timeout. The result would have about 400 million bits. The computation takes somewhere between minutes and hours. Your watchdog needs the GIL to wake up, and it never gets it.

The faulthandler dump made it concrete, and it was worse than the issue described:

Thread 0x16d1f3000 (worker):
  File "local_python_executor.py", line 753 in evaluate_binop   <- computing the pow

Thread 0x1f00d5e80 (main):
  File "threading.py", line 999 in start
  File "concurrent/futures/thread.py", line 180 in submit       <- never returned!

The main thread was still stuck inside ThreadPoolExecutor.submit. It never even reached future.result. The 2 second timer never armed at all.

The existing MAX_OPERATIONS guard (10 million AST operations) does not help either. This is a handful of AST nodes. The entire cost lives inside one of them.

The Sentry angle: monitoring for absence

Entry #1 was about a noisy failure. This bug is the opposite. I pointed a fresh Sentry project at a simulated agent run on the unpatched PyPI release (1.26.0) and got the most unsettling result possible: nothing. No event, no transaction, an empty project. The process was frozen mid-transaction and the SDK never got a chance to flush.

The lesson: for freeze-class bugs you need a supervisor. I added a small watchdog process that gives the worker 25 seconds, then kills it and reports what it saw, with the worker's faulthandler stack attached as evidence:

watchdog: starting worker (before) with 25s budget
worker: smolagents 1.26.0
worker: step 1 executing 'result = 10 ** 10 ** 8'...
watchdog: worker FROZE, killed it, reporting to Sentry

The resulting Sentry issue carries the whole story in one place: environment tagged before, handled by the supervisor, with the frozen frame (evaluate_binop, line 753 of local_python_executor.py) sitting in the attached worker_faulthandler_stack extra.

Sentry's Seer ran root cause analysis on that issue and independently landed on the same conclusion: signal and thread interruption need the GIL, and a single C-level big-int operation never releases it.

Seer's verdict, verbatim:

CodeAgent's 2s timeout uses Python signal-based interruption, which cannot fire during uninterruptible C-level big-int operations that hold the GIL. [...] A single large big-integer arithmetic operation runs entirely as one C-level call that holds the GIL continuously without yielding. CPython cannot deliver signals or switch threads during an uninterruptible C extension call, so no timeout callback fires for the duration of that operation.

The fix: you cannot interrupt it, so refuse to start it

Killing the computation mid-flight is impossible from Python. But predicting the damage is O(1). Before executing **, << or * on integers, the executor now estimates the result's bit length from the operands' bit lengths:

if op == "**":
    estimated_bits = left.bit_length() * right       # upper bound
elif op == "<<":
    estimated_bits = left.bit_length() + right
elif op == "*":
    estimated_bits = left.bit_length() + right.bit_length()

Above 1 million bits (about 300k digits, still generous) it raises an informative InterpreterError:

Operation '**' would produce an integer of around 400000000 bits, exceeding
the maximum of 1000000 bits allowed. Use smaller operands, or
pow(base, exp, mod) for modular exponentiation.

That message matters. The agent surfaces it to the model, and the model can actually act on it: use pow(base, exp, mod), which stays unrestricted because modular exponentiation is fast and legitimate. The agent recovers on the next step instead of hanging the host.

The after run on the patched build: the guard rejects the expression in 0.0 seconds, the error lands in Sentry as a normal actionable issue and step 2 executes fine.

watchdog: starting worker (after) with 25s budget
worker: smolagents 1.27.0.dev0
worker: step 1 error surfaced to the model: InterpreterError: ...
worker: step 2 executing '2 + 2'...
worker: step 2 ok
worker: DONE
watchdog: worker exited with code 0

A robot reviewed my robot fix

Minutes after I opened the PR, OpenAI's Codex reviewer flagged a real hole: my guard checked type(x) is int, which lets bool and int subclasses slip through. True << 10**9 and class BigInt(int) still reached the uninterruptible C calls. Fixed with isinstance, added both as regression tests. AI found the bug class, AI fixed it, AI reviewed the fix. I just steered.

Numbers

Freeze reproduced at 20+ seconds (would have run for hours), external kill required
Fix rejects the same expression in 0.0 seconds
9 explosive patterns blocked: **, <<, chained *, all augmented forms, pow(a, b), bool and int subclass variants
10 legitimate operations verified untouched: 100! via repeated *=, pow(7, 2**64, 97), float pow, 1 ** 10**9
19 new tests, full test file 416 passed, ruff clean
The blocked-pattern tests hang forever on unpatched main. I verified that the honest way, with a stash and a kill switch.

Links

The scariest bugs are not the ones that page you at 3am. They are the ones that make sure nothing ever pages you at all. Instrument for silence.

I fixed a smolagents bug that confused everyone who hit it (with Sentry watching the whole time)

Himanshu Kumar — Thu, 16 Jul 2026 05:31:48 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

I picked huggingface/smolagents, the 28k+ star agent framework where agents literally think in Python code. It had a bug open since issue #1108 that anyone combining MCP tools with agent serialization would eventually slam into.

Call agent.to_dict() on a CodeAgent holding MCP tools (or save() or push_to_hub(), same path) and you get this beauty:

ValueError: Tool validation failed for MCPAdaptTool:
Parameters in __init__ must have default values, found required parameters: name, description, inputs, output_type
- forward: Name 'func' is undefined.
- forward: Name 'mcp' is undefined.

func is undefined? mcp is undefined? I never wrote a forward method. If you hit this in the wild you'd have zero idea what you did wrong. Spoiler: you did nothing wrong.

Bug Fix or Performance Improvement

smolagents serializes tools by reconstructing standalone Python source for the tool class. Tool.to_dict calls validate_tool_attributes() which does static AST analysis, then instance_to_source() so Tool.from_code can rebuild the tool later from source alone.

That contract can never hold for MCP tools. MCPAdaptTool is generated at runtime by mcpadapt inside a closure. Its __init__ takes required parameters and its forward closes over the live MCP client session (func, mcp, logger...). The tool's actual behavior lives on the MCP server, not in Python source. There is no source to reconstruct, so the AST validator chokes on a class that was never meant to pass it.

The interesting part: Tool.to_dict already fails fast with a clear message for three other runtime generated wrappers (Spaces, LangChain, Gradio). MCP tools were just missing from that guard.

Sometimes the right fix isn't making the impossible possible, it's failing loudly and helpfully. Recreating a live MCP session from serialized state would mean silently re-establishing server connections with credentials and trust decisions the library has no business making. So I extended the existing guard to detect MCP tools and raise this instead:

ValueError: Cannot serialize MCP tool 'echo_tool': it wraps a live MCP server session, which cannot be
saved as standalone code. Remove MCP tools from your agent before calling to_dict, save or push_to_hub,
and recreate them with MCPClient or ToolCollection.from_mcp when loading the agent.

From "what is func" to "here's exactly what to do instead" in one guard clause.

Code

Raise informative error when serializing MCP tools #2528

himanshu748 posted on Jul 14, 2026

Fixes #1108

Problem

Calling to_dict() (and therefore save() or push_to_hub()) on an agent that holds MCP tools crashes with a confusing internal error:

ValueError: Tool validation failed for MCPAdaptTool:
Parameters in __init__ must have default values, found required parameters: name, description, inputs, output_type
- forward: Name 'func' is undefined.
- forward: Name 'mcp' is undefined.
...

Reproduction (stdio MCP server, same shape as the tests in tests/test_mcp_client.py):

from mcp import StdioServerParameters
from smolagents import CodeAgent, InferenceClientModel
from smolagents.mcp_client import MCPClient

server_parameters = StdioServerParameters(command="python", args=["-c", echo_server_script])
with MCPClient(server_parameters) as tools:
    agent = CodeAgent(model=InferenceClientModel(), tools=list(tools))
    agent.to_dict()  # ValueError: Tool validation failed for MCPAdaptTool: ...

Root cause

Tool.to_dict serializes a tool by reconstructing standalone source code for its class: it calls validate_tool_attributes(self.__class__) and instance_to_source(...) so that Tool.from_code can later rebuild the tool from that source alone.

That contract cannot hold for MCP tools. MCPAdaptTool is generated at runtime by mcpadapt, its __init__ takes required parameters and its forward is a closure over the live MCP client session (func, mcp, logger, ...). The tool's behavior lives on the MCP server, not in Python source, and the underlying connection is not serializable, so source reconstruction fails validation with the cryptic error above.

Tool.to_dict already fails fast with a clear message for the other three runtime generated wrapper classes (SpaceToolWrapper, LangChainToolWrapper, GradioToolWrapper). MCP tools were missing from that guard.

Fix

Extend the existing guard in Tool.to_dict to detect MCP tools and raise an actionable error:

ValueError: Cannot serialize MCP tool 'echo_tool': it wraps a live MCP server session, which cannot be
saved as standalone code. Remove MCP tools from your agent before calling to_dict, save or push_to_hub,
and recreate them with MCPClient or ToolCollection.from_mcp when loading the agent.

Detection matches the runtime class name, following the existing convention in the same block, since mcpadapt is an optional dependency. The from_dict direction needs no change: serialization now fails fast with a clear message, and recreating a live MCP session is a user decision (server lifecycle, credentials, trust) that Tool.from_code could never perform safely.

A note documenting the limitation is added to the MCP section of the tools tutorial.

Tests

Two tests in tests/test_mcp_client.py, using the existing echo_server_script stdio fixture:

test_mcp_tool_to_dict_raises_informative_error: tool.to_dict() raises the clear error.
test_agent_to_dict_with_mcp_tool_raises_informative_error: CodeAgent.to_dict() raises the clear error (the exact scenario from the issue).

Both fail on main with the old Tool validation failed for MCPAdaptTool error and pass with this change. make quality passes. tests/test_mcp_client.py (7 passed), tests/test_tools.py and the agent serialization tests in tests/test_agents.py pass locally; the two pre-existing failures in test_integration_from_mcp_with_streamable_http and test_integration_from_mcp_with_sse also fail on a clean main checkout (local port binding) and are unrelated.

View on GitHub

PR: https://github.com/huggingface/smolagents/pull/2528 (Fixes #1108)

Two new tests using the existing stdio echo server fixture, both fail on main and pass with the fix. make quality clean. Docs note added to the MCP tools tutorial so nobody has to learn this the hard way again.

My Improvements

Users hitting this now get an actionable error instead of AST validator internals
The fix follows the repo's existing convention exactly (same guard block, same style as the Space/LangChain/Gradio cases), which is what makes a one-commit PR actually mergeable
Documented the limitation where users would look for it
Regression tests covering both the raw tool and the full CodeAgent scenario from the original issue

Best Use of Sentry

This is where it gets fun. I built a small demo agent app (a weather checkpoint agent using my patched smolagents with an MCP tool) and wired in the Sentry Python SDK with error monitoring, tracing and AI agent monitoring before touching the fix.

Step 1: catch the crash. Running the demo on unpatched smolagents, the cryptic ValueError landed straight in Sentry as an unhandled issue with the full 20+ line "undefined name" spam captured.

Step 2: let Seer take a shot. I ran Seer root cause analysis on the captured issue. Its diagnosis, fully independent of my PR:

MCPAdaptTool is a dynamically generated inner class created by mcpadapt's SmolAgentsAdapter.adapt() closure, so its init has required parameters and its methods reference closure variables that are not visible as class-level attributes... making it fundamentally incompatible with smolagents' static source-code validation.

Which is, almost line for line, the root cause I wrote in the PR. AI-assisted debugging where the AI and the human converge on the same diagnosis independently is exactly the confidence check you want before shipping a fix upstream.

Step 3: verify and resolve. Same demo on the patched version runs clean, agent traces show the gen_ai spans (invoke_agent, execute_tool) nested under the workflow and the Sentry issue is marked resolved.

Used the bugsmash26 code for the $100 credits too. Thanks Sentry 🛹

What I learned

Serialization boundaries are where abstractions leak. smolagents' "tools are source code" model is elegant right up until a tool is actually a live network session wearing a Tool costume. The mature move for a library isn't to pretend otherwise, it's to name the limitation clearly at the exact moment the user hits it.

Also: watching Seer independently arrive at your root cause is a genuinely great feeling. Like a second engineer nodding at your RCA.

Built during DEV's first Summer Bug Smash. Find me on GitHub @himanshu748 or X @jhahimanshu653.

I Traced a Multi-Step LLM Agent With Self-Hosted SigNoz. One Feature Sold Me.

Himanshu Kumar — Sat, 11 Jul 2026 08:22:06 +0000

Multi-step LLM agents fail in a way normal backends don't. Nothing crashes. The pipeline "works", the answer is just bad, slow or three times more expensive than yesterday. print() debugging tells you nothing, because the interesting question is never "did step 3 run". It is "what did step 3 see, which model actually answered and what did it cost".

So I self-hosted SigNoz and pointed a simulated agent pipeline at it: a four-step research assistant (plan, retrieve, generate, synthesize) instrumented with OpenTelemetry, emitting traces, metrics and logs, with GenAI semantic-convention attributes (gen_ai.request.model, gen_ai.usage.input_tokens and friends) on every LLM call.

This post is about the feature that turned out to be the most useful. It was not the one I expected.

Setup: one CLI, eight lines of YAML

Self-hosting used to mean wrangling a long docker-compose file. SigNoz now ships Foundry, a small CLI that casts the whole stack:

curl -fsSL https://signoz.io/foundry.sh | bash   # installs foundryctl (checksum-verified)
foundryctl cast -f casting.yaml                  # deploys the full stack on Docker

with a casting.yaml that is all of eight lines:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker

A few minutes of image pulls later, ClickHouse, Postgres, the SigNoz backend and an OTel collector were running. The UI is on localhost:8080 and the collector listens on 4317 (gRPC) and 4318 (HTTP).

One gotcha worth knowing: telemetry is rejected until you create the admin account in the UI. The collector registers with the backend over OpAMP, and until an organization exists the backend answers "cannot create agent without orgId" and the OTLP ports reset every connection. If your exporter logs Connection reset by peer on a fresh install, you haven't finished the two-minute signup at localhost:8080 yet. Create the account and ingestion starts working within about thirty seconds, no restarts needed.

My demo app needed zero SigNoz-specific code: the stock OpenTelemetry SDK exporting OTLP to localhost:4318. That is the point of an OTel-native backend. There is no vendor agent, so nothing about the app knows SigNoz exists.

The pipeline being observed

Each simulated request produces one trace:

agent.request                    (root, 2.02s)
├── agent.plan                   (850ms)
│   └── gen_ai.generate plan     (696ms, gen_ai.* attributes)
├── agent.retrieve               (461ms, sometimes errors: vector store timeout)
├── agent.generate               (564ms)
│   └── gen_ai.generate answer   (564ms, gen_ai.* attributes)
└── agent.synthesize             (139ms, sometimes errors: citation validation)

Every gen_ai.generate span carries the GenAI semantic conventions: operation name, provider, model, input tokens and output tokens. Alongside the traces the app emits counters for token usage (gen_ai.client.token.usage) and estimated spend (agent.llm.cost), both tagged by model and provider, a request-duration histogram and structured logs that inherit the active trace context automatically.

Sixty simulated requests later: 420 spans, 134 logs and a few hundred metric samples, all visible in the UI. The Services page picked up research-assistant on its own with RED metrics already computed (p99 latency, error rate, throughput). I wrote no configuration for that.

The feature I expected to love: the trace waterfall

And it is genuinely good. The trace detail view renders a flame graph and waterfall that read exactly like the agent's mental model: plan, then retrieve, then generate, then synthesize, with the LLM calls nested inside the steps that made them. Clicking any gen_ai.generate span opens a details panel with every attribute I set: model llama-4-maverick, provider meta, 1501 input tokens, 90 output tokens, plus a percentile badge telling me this span sat at p10 of its peers. "Why did this request take four seconds" stops being a mystery and becomes a picture.

But a pretty waterfall is table stakes for a tracing tool. The thing that sold me was what happens around it.

The feature I actually loved: your attributes become the query language

Here is the moment it clicked. In the Trace Explorer I typed:

gen_ai.request.model = 'qwen3-coder-plus'

Two things happened. First, the autocomplete suggested qwen3-coder-plus before I finished typing, because SigNoz had already indexed the values of an attribute I invented an hour earlier. Second, the results came back instantly: only the LLM spans that were served by that model, across every trace in the system.

Stop and consider what that means for agent debugging. I never told SigNoz what gen_ai.request.model is. There is no schema registration, no field mapping, no config file. Any attribute your instrumentation emits is immediately a first-class, autocompleted, indexed query dimension. Your instrumentation vocabulary becomes the product's vocabulary. For LLM systems, where all the interesting facts live in custom attributes (model, provider, token counts, agent role, tool name), this is the difference between an observability tool that fits and one you fight with.

And the same query keeps working as you move across signals:

Span to logs, one click. From any span's details panel, the Logs tab jumps into the Logs Explorer with trace_id = '<this trace>' pre-filled and the time window auto-scoped. I got back exactly the two log lines belonging to that request, correlated purely by the trace context the OTel logging handler injects. Nobody parses log lines to find a request id. The correlation is structural.
Query to dashboard or alert, two clicks. Every explorer view has "Add to Dashboard" and "Create an Alert" buttons at the bottom. The query I used to investigate becomes the panel that monitors and the alert that pages, without re-expressing it in a different query language. The investigate-then-monitor loop is one surface, not two tools.
Metrics inherit the same attributes. The Metrics Explorer summary listed my custom metrics with their types and units the moment they arrived: gen_ai.client.token.usage showing 6 time series (3 models times 2 token types), agent.llm.cost in usd showing 3. Cost per model is a group-by away, using the exact attribute names from my instrumentation.

Most observability stacks treat custom attributes as second-class blobs that need schema work before they are queryable. SigNoz treats them as the whole point. For agent systems, they are.

What else is in the box

Going deep on one feature meant walking past a lot of others. Quick notes from the tour, agent-flavored:

Services (APM) pages, zero config. Any service that sends spans gets RED metrics automatically: request rate, error rate, latency percentiles. In a multi-agent system every agent that traces becomes its own monitored service for free.
Dashboards. Panel grids built with the same query builder, so traces, logs and metrics coexist on one board, importable and exportable as JSON. A "cost per model per agent" board takes minutes because the query language is the one you already know.
Alerts with notification channels. Threshold or anomaly rules on any query, delivered to Slack, PagerDuty or a plain webhook. That webhook option is quietly powerful for agent systems: an alert can call your own service and close the loop from observability back into behavior, e.g. demoting a flaky model.
Trace Funnels (beta). Define a sequence of span conditions and measure conversion and latency between the steps across all traces. For a pipeline like mine that is literally a success funnel: how many requests that planned also retrieved, generated and synthesized cleanly.
Exceptions. Recurring span errors cluster into groups instead of being scattered across individual traces, so "vector store timeout" is one row with a count, not forty needles in a haystack.
Logs Pipelines. Collector-side log pre-processing (parse JSON, extract or drop fields) before storage, for when you can't change the code that emits the logs.
Metrics summary as a cardinality watchdog. The same inventory view that showed my metric types also shows time-series counts per metric, which is where you catch a label-explosion mistake before it hurts.

Each of these uses the attribute vocabulary your instrumentation defines. That is the theme of the whole product, and it is why the one feature I picked is really the foundation the rest stand on.

What this means if you're building agents

The uncomfortable truth about multi-agent LLM systems is that their most important behavior (which model ran, what it consumed, what it cost, why it was retried) is invisible to conventional monitoring. It lives entirely in domain-specific span attributes. A backend that makes those attributes instantly queryable, correlatable across traces and logs, and promotable into dashboards and alerts is not a nice-to-have there. It is the debugger.

The whole experiment cost me an afternoon: one CLI install, eight lines of YAML, a stock OTel SDK and zero vendor code in the app. I'm taking this setup into the Agents of SigNoz hackathon (July 20 to 26), where the plan is considerably less simulated. If you're building anything agent-shaped, self-host SigNoz and type one of your own attribute names into the Trace Explorer. That autocomplete dropdown is the moment you'll get it too.

The demo app is a 170-line Python script using only opentelemetry-sdk and the OTLP HTTP exporter. Stack: SigNoz self-hosted via Foundry on Docker (ClickHouse + Postgres + OTel collector), macOS host.

Demo code, self-host config and blog source: github.com/himanshu748/signoz-agent-observability. Team 404Found.