DEV Community: Himanshu Kumar

Tracing a multi-agent LLM system: otel-swarm and a SigNoz dashboard pack

Himanshu Kumar — Sun, 26 Jul 2026 17:40:05 +0000

The problem

A single LLM call is easy to reason about. You send a prompt, you get tokens back, you log the latency and you move on.

A swarm is not that. Four or five agents run, some in parallel, each with its own model. One of them times out and quietly falls back to a cheaper model. A critic agent reads the output and sends work back for another round. When the whole thing takes 40 seconds instead of 12, or costs three times what you budgeted, you have no idea which agent did it. Your logs are a flat stream of interleaved lines from concurrent tasks, with no parent-child structure and no way to ask "which role burned the tokens".

Traces solve this exactly. The catch is that hand-wiring OpenTelemetry across every role, every provider call and every retry path is boring work that nobody wants to do twice, and if you skip a level the trace tree lies to you.

So I extracted the instrumentation out of DevSwarm (a multi-agent code generator I built for this hackathon) into a standalone MIT library: otel-swarm.

The API

One createSwarm() call, then three verbs.

import { createSwarm } from 'otel-swarm';

const swarm = createSwarm({ service: 'my-swarm', otlpEndpoint: 'http://localhost:4318' });

await swarm.task('generation', { 'my.prompt': prompt }, async (root) => {
  const plan = await swarm.agent('planner', () =>
    swarm.llm('planner', {
      model: 'primary-model-id',
      fallbackModel: 'fallback-model-id',
      call: async (model) => {
        const res = await yourProviderCall(model);
        return { content: res.text, inputTokens: res.usage.in, outputTokens: res.usage.out };
      }
    })
  );

  await swarm.agent('critic', async (span) => {
    swarm.reviewEvents(span, issues);   // each issue becomes a critic_catch span event
  });
});

swarm.events.on('event', (e) => { /* llm_start, llm_end, fallback, critic_catch */ });

task() opens the root span, agent() opens a child span per role and llm() opens a GenAI-semconv span per model call. call(model) is invoked again with fallbackModel if the primary throws, and the switch is recorded as a fallback_promotion span event carrying from, to and the verbatim provider error.

The last line matters more than it looks. swarm.events is an EventEmitter that mirrors the span lifecycle, and LLM events carry the traceId. A live UI reads the emitter, the tracing backend reads the OTLP exporter and both are fed from the same code path, so the dashboard and the UI can never disagree about what happened. The traceId means a row in your UI deep-links to the exact trace in SigNoz.

Running npm run example against a local SigNoz produced 9 spans under service otel-swarm-demo: a root generation span of 1325ms, then agent.planner/llm.planner, agent.frontend/llm.frontend, agent.backend/llm.backend and agent.critic/llm.critic. llm.frontend carried a fallback_promotion event and agent.critic carried a critic_catch event. The whole story is one trace.

In production use inside DevSwarm the same library has traced 29 generations, 243 model calls across 8 models, 3.78 million tokens and 257 review catches.

Lesson 1: put the fallback in a span event, never on the attribute you group by

This one cost me an afternoon.

An earlier version of the library did the obvious thing when a primary model failed: it overwrote gen_ai.request.model with the fallback's name, so the span would "tell the truth" about which model actually answered.

That is wrong, and it is wrong in a way that is hard to see. Every dashboard panel that groups by model then attributes the primary's failure, its timeout and all of its wasted latency to the fallback that cleaned up after it. My "tokens and latency by model" table showed a cheap fallback model with terrible p95 latency and my expensive primary looking flawless, because every time the primary blew up its cost was silently reassigned to whoever picked up the pieces. I spent an afternoon convinced the wrong model was slow.

The rule that comes out of it: the attribute you group by must record the model you attempted, not the model that ended up answering. The promotion is a discrete thing that happened during the span, and a discrete thing that happened during a span is what span events are for:

span.addEvent('fallback_promotion', { from: model, to: fallbackModel, reason: String(err.message || err) });

Now the grouped panel keeps attributing failure to whoever caused it, and "how often does this role get promoted" is a separate query over events. Two different questions, two different storage locations, no cross-contamination.

The general form: if a value can change mid-span, it does not belong on an attribute you aggregate over.

Lesson 2: reading inside span events in ClickHouse is not obvious

The consequence of putting things in events is that you now have to query events, and SigNoz stores them as an array of JSON strings on the span row. There is no autocomplete that will lead you here. The pattern is ARRAY JOIN to flatten the array into one row per event, then JSONExtractString to reach into the event's attributeMap:

SELECT JSONExtractString(e, 'attributeMap', 'severity') AS severity,
       JSONExtractString(e, 'attributeMap', 'target') AS target,
       count() AS catches
FROM signoz_traces.distributed_signoz_index_v3
ARRAY JOIN events AS e
WHERE serviceName = '{{.service}}' AND name = 'agent.critic'
  AND JSONExtractString(e, 'name') = 'critic_catch'
GROUP BY severity, target ORDER BY catches DESC

That gives you review findings broken down by severity and by what they were aimed at, straight out of span events, with no separate metrics pipeline.

When you only need to know whether an event fired at all, do not pay for the ARRAY JOIN. arrayExists with a substring test over the raw array is enough and it keeps one row per span, which is what you want for a time series:

SELECT toStartOfInterval(timestamp, INTERVAL 15 MINUTE) AS ts,
       attributes_string['swarm.role'] AS role,
       count() AS value
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = '{{.service}}'
  AND arrayExists(e -> e LIKE '%fallback_promotion%', events)
  AND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
GROUP BY ts, role ORDER BY ts

The dashboard pack

The repo ships 3 importable SigNoz dashboards (POST each JSON to /api/v1/dashboards). Every query uses a {{.service}} dashboard variable that defaults to otel-swarm-demo, so you point the pack at your own service by editing one dropdown instead of doing find-and-replace across query strings.

Generation Overview answers "is the swarm healthy right now": generation count, review-gate pass rate, seconds per generation over time and a per-role table of calls, average latency, p95 and errors.
LLM Economics answers "where did the tokens go": total tokens, tokens over time by role, a tokens-and-latency table by model and fallback promotions by role (the arrayExists query above).
Review-Gate Funnel answers "is the reviewer doing its job": total catches, regeneration rounds, catches per generation over time and a table of recent generations with verdict, catches and regenerations.

The alert rules

Two rules ship in the v2alpha1 schema (POST to /api/v2/rules).

The first is a fallback-promotion spike. If promotions jump, a provider is degrading and you are silently paying a different bill than you planned.

The second is the one I care about more: a review catch-rate flatline. If your critic agent suddenly stops finding anything, the tempting read is that the generators got better. In practice the reviewer broke: a prompt change made it answer in a shape the parser drops, or its model started returning empty content and the failure is being swallowed as "no issues found". A quality gate that passes everything is indistinguishable from no quality gate, and it fails silently by construction. Alert on the absence.

The repo also includes casting.yaml and casting.yaml.lock, the Foundry config the SigNoz instance behind these dashboards was installed from, so the backend the pack targets is reproducible rather than assumed.

Install and run

npm install github:himanshu748/otel-swarm

To see the example trace end to end against a local SigNoz:

git clone https://github.com/himanshu748/otel-swarm && cd otel-swarm
npm install
npm run example                                                     # spans to console
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npm run example   # spans to SigNoz

With no endpoint set it exports to the console, so you can check the span tree before wiring a backend.

Closing

The library is about 130 lines. Most of the value is not code, it is the two decisions above: keep mutable facts in span events, and treat "the reviewer found nothing" as a symptom rather than a result. Both were learned by getting them wrong first.

I built otel-swarm with Claude Code, including the dashboard JSON and the ClickHouse queries in this post.

Repo: github.com/himanshu748/otel-swarm, MIT.

We instrumented an AI agent swarm with SigNoz, and its own telemetry told us we were wrong about almost everything

Himanshu Kumar — Sat, 25 Jul 2026 15:01:20 +0000

Built for the WeMakeDevs Agents of SigNoz hackathon, July 2026.

Mission Control. The graph is the swarm, the river underneath it is the live span stream, and every bar deep-links into that trace in SigNoz.

DevSwarm turns one prompt into a working full-stack app. Five open-weight models plan it, build it, review it and repair their own routing. Nothing it produces is trusted blindly, and every step is an OpenTelemetry span in SigNoz, including the steps that go wrong.

We built the observability first, expecting it to prove the thing worked.

It did something more useful. It spent a week proving that almost everything we believed about our own system was wrong. We blamed a model for a limit we had set ourselves. We blamed a provider outage on the model. We assumed our review agent was our strongest link when it was measurably the weakest. And we spent days writing a design system that, when we finally measured it, was making the output worse.

Not one of those was found by reading the code again. Every single one came off a span event, a dashboard row or a benchmark that disagreed with us.

So this is not an architecture post. It is six times the telemetry told us we were wrong, with the queries.

The current numbers, all read live out of SigNoz rather than typed into a slide: 22 generations, 189 traced model calls across 8 models, 2.85 million tokens, 225 critic catches, 19 fallback promotions and 24 generated apps each reporting under their own service name.

What the swarm actually is

Five roles, each on the open-weight model that measured best for that job:

role	model	job
planner	GLM-5.2	turn a prompt into a typed build plan and a locked API contract
frontend	GLM-5.2	one self-contained index.html against that contract
backend	Qwen3-Coder-480B	one Express server against the same contract
critic	Kimi-K2.7-Code	review both, gate the merge, route catches back to their owner
doctor	GLM-5.2	read the swarm's own traces and repair its model routing

Every number on our own landing page is a live ClickHouse query against the trace store. Marketing copy that drifts from the telemetry is impossible by construction.

Everything is served through Hugging Face Inference Providers. There are zero closed-model API calls in the system, which turned out to matter for reasons we did not anticipate (see the provider section below).

The critic is the load-bearing part. Frontend and backend are generated in parallel from the same contract, then an independent model reviews both for contract conformance, security and runtime bugs. Real catches route back to the agent that owns them, that agent patches its own file and the critic re-reviews only the delta. Two regeneration rounds, then it ships with an honest verdict either way.

The same gate runs on changes. Asking a finished app for "a star rating on each book, settable when adding one" re-plans against the stored contract, re-runs only the agents that instruction touches, and puts the result through the identical review. A refinement is its own root span, so a change request is as traceable after the fact as the build that preceded it.

Why we instrumented before we polished

A multi-agent system fails in ways a single-model tool does not. A call can succeed while producing an unusable artifact. A fallback can rescue a request so smoothly that nobody notices the primary is dead. Latency can triple because one role silently started thinking twice as long. None of that shows up in a request log.

So the very first thing that worked in this project was not code generation. It was a trace.

SigNoz is self-hosted through Foundry, which is a single-config install. Our casting.yaml and casting.yaml.lock are committed to the repo so the deployment is reproducible by anyone, judges included.

Three signals, and what each one carries

Traces. Every model call is a span named llm.<role> carrying GenAI semantic conventions:

span.setAttributes({
  'gen_ai.operation.name': 'chat',
  'gen_ai.request.model': model,
  'gen_ai.usage.input_tokens': usage.prompt_tokens,
  'gen_ai.usage.output_tokens': usage.completion_tokens,
  'devswarm.role': role
});

Two span events do the heavy diagnostic lifting. fallback_promotion records that a primary failed, which model took over and the verbatim reason. critic_catch records every issue the review agent found, with its target and severity. Both are events rather than separate spans on purpose: they belong to the call they describe, and they survive in the trace even when the call ultimately succeeds.

Metrics. Six counters and a histogram, because some questions are time series questions rather than trace questions: devswarm.tokens, devswarm.llm.calls, devswarm.llm.duration, devswarm.fallback.promotions, devswarm.critic.catches, devswarm.generations, devswarm.refinements. Labelled by role, model and outcome.

Logs. Structured records for the things a human reads during an incident: a fallback promoting, a doctor diagnosis, a generation completing with its verdict and catch count. Same resource attributes as the traces, so a log line and a span line up.

The whole trace layer is now extracted into a small library, otel-swarm, that any multi-agent system can drop in. DevSwarm consumes it as a real dependency, which means if the library breaks, our own dashboards go dark first. That felt like the honest way to ship it.

One generation as a flame graph. Planner, then frontend and backend in parallel, then the critic.

The log stream during a rough run. WARN lines are fallback promotions, each naming the model that failed and why.

Reading the swarm out of ClickHouse

Five dashboards, all committed as JSON in observability/dashboards/. The one we actually live in is Command Center: is the swarm healthy, and if not, which role. The odd one out is Born Observable, which contains no swarm data at all, only the generated apps reporting under their own service names.

Three queries worth sharing, because span events in ClickHouse are not obvious the first time.

Role health, straight off the trace store:

SELECT attributes_string['devswarm.role'] AS role,
       count() AS calls,
       countIf(statusCode = 2) AS errors,
       round(quantile(0.95)(durationNano) / 1e9, 1) AS p95_s,
       sum(attributes_number['gen_ai.usage.input_tokens']
         + attributes_number['gen_ai.usage.output_tokens']) AS tokens
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'devswarm' AND name LIKE 'llm.%'
GROUP BY role ORDER BY calls DESC

Fallback promotions over time, which requires reaching into the events array:

SELECT toStartOfInterval(timestamp, INTERVAL 30 MINUTE) AS ts,
       attributes_string['devswarm.role'] AS role,
       count() AS value
FROM signoz_traces.distributed_signoz_index_v3
WHERE serviceName = 'devswarm'
  AND arrayExists(e -> e LIKE '%fallback_promotion%', events)
GROUP BY ts, role ORDER BY ts

The one that took longest to work out reads inside the events, so you can ask what the review gate actually caught rather than how many times it caught something:

SELECT JSONExtractString(e, 'attributeMap', 'severity') AS severity,
       JSONExtractString(e, 'attributeMap', 'target') AS target,
       count() AS catches
FROM signoz_traces.distributed_signoz_index_v3
ARRAY JOIN events AS e
WHERE serviceName = 'devswarm' AND name = 'agent.critic'
  AND JSONExtractString(e, 'name') = 'critic_catch'
GROUP BY severity, target ORDER BY catches DESC

The answer, over the week: 221 high, 46 medium, 10 low, and the frontend agent is on the receiving end of 70 percent of them. That single row changed how we think about the swarm. The half of the system generating markup and client-side state is where the bugs live, not the half touching the database.

One design decision we are glad about: the marketing numbers on our own landing page are fetched from these same queries at request time. The page cannot drift from the telemetry, because there is only one source of both.

Command Center. Top row answers "is the swarm healthy", the role-health table answers "which role", while the fallback chart should trend to zero.

LLM Economics: tokens and latency per role and per model, which is how we caught the frontend role burning two thirds of its budget on reasoning.

What one app actually costs

Every token in the table below came off a span. This is one real run, the letterpress site above, priced at the rates the Hugging Face router itself reports for the providers we use.

role	model	in	out	cost
frontend	GLM-5.2	27,855	38,888	$0.2101
critic	Kimi-K2.7-Code	50,996	14,539	$0.1066
planner	GLM-5.2	670	4,090	$0.0189
backend	Qwen3-Coder-480B	2,800	3,778	$0.0069
		82,321	61,295	$0.34

Thirty four cents for a designed marketing site with a working Express backend, a waitlist that validates email and rejects duplicates, plus its own OpenTelemetry wiring. Across passing runs the range is about 18 cents to 56 cents.

Two things in that table surprised us.

The critic costs fifteen times what the backend author costs. Reviewing the code is dramatically more expensive than writing it, because review means reading both artifacts in full, twice, while the backend agent writes one file once. Nobody budgets for that. If you are building a review gate into an agent system, it is not a rounding error on top of generation, it is a third of your bill.

And a failed run costs more than a successful one. Our worst generations burned 232,000 tokens hitting the regeneration ceiling, against 74,000 for the cleanest pass. So convergence is not only a quality metric, it is the cost metric. Fixing the contract-format bug in finding five did more for our unit economics than any model swap we made.

Alerts that wake an agent instead of a human

This is the part of the build we are proudest of, and it is a genuinely small amount of code.

Two alert rules live in observability/alerts/: a fallback-usage spike and a critic catch-rate flatline. Both notify a webhook channel called swarm-doctor, which points at POST /api/doctor/webhook on the swarm itself.

When an alert fires, the Doctor wakes up, queries the swarm's own traces for the last hour and decides what to do about the routing table. It promotes a backup model, resets a recovered primary or does nothing, then explains itself in plain English using the numbers it just read. Its first real diagnosis, verbatim:

"The critic role is the clear problem area: its primary triggered 6 fallback promotions out of 9 calls (67%) with a 22% error rate and p95 latency of 242s. Backend, planner and doctor are healthy."

The Doctor's own model calls are traced too, so the healer is exactly as observable as the patient.

The Doctor reporting a healthy swarm. It read 180 minutes of its own traces to say so, and it is honest about sample size: "call volume is very low, so latency figures are not statistically meaningful".

Two hard-won SigNoz API notes, since we lost hours to both:

Alert rules must be created against /api/v2/rules with schemaVersion: v2alpha1, a notificationSettings block and at least one channel. The v1 endpoint accepts the request and returns "alert rule is not valid" with no indication of which field is wrong. Dashboards, by contrast, go to /api/v1/dashboards with a SIGNOZ-API-KEY header and behave exactly as documented.

Also: a cold Docker restart can leave ClickHouse replicas read-only until Keeper reconnects. It usually self-heals within a minute. If it does not, SYSTEM RESTORE REPLICA per table clears it.

Apps that are born observable

Every app the swarm generates ships instrumented. Alongside index.html and server.js, each generated folder gets an otel.mjs bootstrap, a package.json, and a signoz-dashboard.json scoped to that app's own service name. If a SIGNOZ_API_TOKEN is configured, the dashboard is created in SigNoz at generation time, before the user has opened the preview.

So the generated app appears in SigNoz as its own service, with RED metrics and a routes table, seconds after it exists. Twenty four of them are in our instance right now.

Worth saying because people assume otherwise: there is no image model anywhere in this pipeline. The swarm generates 227 inline SVG elements across the 26 apps it has built, an average of 8.7 per app, and every one of them was written as markup by a language model. The Vandercook press in the screenshot above is hand-drawn SVG, not a generated image. The only assets we ever image-generated are DevSwarm's own favicon and social card, which are branding for the tool rather than anything the swarm produces.

One of the outputs. The Vandercook press is inline SVG the model drew itself, and this app reports to SigNoz under its own service name from the moment it boots.

One caveat we had to learn the hard way: an app only emits spans while its backend is actually running. Early on our preview served the frontend statically, so the generated Express server never booted and the app silently fell back to localStorage. The preview looked perfect and the service page was almost empty. That mismatch, two spans where there should have been dozens, is what gave the bug away. Previews now spawn the real server as a child process and proxy to it.

Six things our own telemetry told us we had wrong

This is the section I would want to read, so it is the longest one.

1. A "model limitation" was a stale constant we wrote ourselves.

Traces showed every frontend failure as finish: length, truncating full-page HTML. We concluded GLM-5.2's provider capped completions at 16384 tokens and moved the role to another model. The cap was real when we found it. It was also in our own config, and when the provider limit later lifted, our constant kept enforcing a limit that no longer existed. The average frontend artifact needs about 19,000 output tokens. We had guaranteed truncation and blamed the model for a fortnight.

2. The real cause was provider roulette, visible only in the span event text.

After removing our own cap, GLM still failed intermittently with a 400: max_completion_tokens is limited to 16384 for glm-5.2. The Hugging Face router load-balances a model across every provider serving it, and their limits disagree. We probed all seven: scaleway caps at 16384, featherless at 32768, novita and zai-org at 131072, while together, fireworks-ai and deepinfra accept 200000 or more. Unpinned, roughly one request in seven hit the strict provider and died instantly. Pinning the model to one provider produced our first ever generation with zero fallbacks. That entire diagnosis came out of the reason attribute on a fallback_promotion event.

3. Our span attribute was hiding the failures we most wanted to see.

When a primary failed, our code overwrote gen_ai.request.model on the span with the fallback's name. It seemed tidy. It meant every dashboard row attributed the primary's failure, and its wasted latency, to the fallback that cleaned up after it. We spent an afternoon convinced the critic's backup model was slow and error-prone. Isolating them in a benchmark showed the opposite: the backup was fine at 7 seconds, and the primary was the problem. If you take one implementation detail from this post, take this one. Record the model you attempted, and put the promotion in an event, not on top of the attribute you will later group by.

4. Our review gate was the weakest model in the swarm.

We had never benchmarked the critic, so we built one: a generated app with three documented contract defects, three runs per model, scored on defect recall. DeepSeek-V4-Pro, our incumbent primary, found 2 of 9. One run burned its entire 32768-token budget and returned nothing parseable. Kimi-K2.7-Code found 8 of 9 and was consistent across runs. The clearest pattern in the data was that on a review task, reasoning volume tracks defect recall: the terse models answered in under 250 output tokens and missed real bugs.

5. A 20 percent pass rate was one missing sentence in the contract.

Our plans specified field names and types but never formats, ranges or nullability. So the backend rejected rating: 0 while the frontend sent 0 as its default, and the backend demanded YYYY-MM-DD while the frontend sent full ISO strings. Three consecutive generations hit the regeneration ceiling on exactly this class of disagreement. The fix was making the planner write a binding rules string per field, for example "integer 0 to 5 inclusive, where 0 means unrated and is a valid value". Both builders now read the same sentence. Catches dropped from 9 to 3 and the next run passed.

6. Our design system was making the output worse.

We wrote a careful design guide so generated apps would not look like generated apps. Then we measured it: same model, same prompt, the only variable being whether the guide was attached. Without it, 14 inline SVGs, 3 animations and a deliberate typeface pairing. With it, 5 SVGs, 1 animation and Courier New. Three of our own rules did that. "System font stack is fine" told the model not to bother choosing type. "Cut any animation that does not serve the subject" read as licence to strip ornament. And our frontend prompt banned all external requests, which silently banned Google Fonts, so it could not have chosen a real typeface even if it wanted to. We had written a list of prohibitions, which is good at preventing bad output and bad at producing good output.

Before. Mono labels from a system stack, and card colours the backend invented at random.

After. Same model, same prompt, three rules removed from our guide: Fraunces display type, a drawn logo mark, filter chips carrying live counts, plus the books rendered as spines on a shelf.

There was a second layer to that one. The guide had a generous section for marketing sites, licensing scroll reveals, entrance sequences and layered depth, plus a stingy section for apps. Every app the swarm built was being held to a deliberately plainer standard than every site, and nobody had noticed because the sites looked great.

How this was built

DevSwarm was built with Claude Code, which is worth stating plainly rather than leaving as an inference. An AI coding agent helped build an AI coding agent, and the hackathon rules ask entrants to declare assistant use, so here it is.

It is also relevant to the point of this post. Every finding in the section above started as a confident, wrong belief held by both of us, human and assistant alike. The stale token cap, the model we blamed for a provider's limit, the review gate we assumed was our strongest link, the design system we were sure was helping. None of those were resolved by reasoning harder about the code. They were resolved by a span event, a dashboard row or a benchmark disagreeing with us.

That is the argument for instrumenting an agent system early. When you are building with agents, and with an agent, the telemetry is the only participant in the conversation with no opinion to defend.

What is worth copying from this build

If you are instrumenting an agent system, three things paid for themselves immediately.

Put the reason text in the span event. Not a code, not an enum, the actual provider error string. Both of our worst bugs were solved by reading that field, and neither would have been visible in a metric.

Never overwrite an attribute you intend to group by. Add, do not replace.

Make your product read its own telemetry. Our landing page statistics, our Doctor's diagnosis and our dashboards all run the same queries against the same trace store. It removes a whole category of drift, and it turns your observability stack from a debugging tool into a feature.

The uncomfortable part

The through line of every finding above is the same, and I have thought about it more than I expected to.

Not one of these was a hard problem. A stale constant. A provider with a different limit. An attribute overwritten in the wrong place. A missing sentence in a contract. Three over-cautious lines in a style guide. Any of them would have been a five minute fix if we had known. Together they cost us most of a week and made the system look, from the outside, like the models were letting us down.

They were not. Every single time, the model did exactly what our configuration told it to do. The failure was always upstream of the model, in something we had written and then stopped looking at.

I think that is the actual lesson of building with agents, and it is not a comfortable one. The debugging skill is not prompt engineering. It is being willing to believe your instrumentation over your own memory of what you configured three days ago. We only got there because the telemetry kept producing numbers that made our explanations impossible.

If you are building something similar, I would genuinely like to know whether your experience matches. My suspicion is that a lot of "the model is not good enough" is actually "my config is stale and I have no way to see it".

The bug that never crashed: how I fuzzed an AI's own code sandbox and found it lying to its model

Himanshu Kumar — Wed, 22 Jul 2026 13:26:43 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The bug that never crashed

The scariest bug I caught this month never threw a stack trace. It never paged anyone. It just quietly made an AI agent dumber, on repeat, and handed the blame to the model.

Here is the whole story, because the way I found it turned out to matter more than any single fix.

Where it started

smolagents is Hugging Face's agent framework, 28k+ stars, and it has an unusual design: the agent writes Python as its reasoning, and a sandboxed interpreter (LocalPythonExecutor) runs that code. The model thinks by coding. So the sandbox is not a side feature. It is the surface the model lives on.

I went in looking for one bug to fix for this challenge. Every obvious open issue was already claimed or had a competing PR. So instead of reading the issue tracker, I did something almost stupid: I fed the sandbox ordinary, boring, valid Python and watched what it refused to run.

The rule I gave myself: if it is valid Python and the sandbox rejects it, that is a bug. The model writes valid Python. If the sandbox chokes on it, the agent pays.

The first false rejection

config = {**{"temperature": 0.7}, "top_p": 0.9}

Merging dicts with **. Every LLM writes this. The sandbox's answer:

InterpreterError: NoneType is not supported.

There is no None in that line. I stared at it, then went to the source. In Python's AST, a **spread entry inside a dict literal has None where the key would be, a marker that says "this is a spread." smolagents walked the keys and tried to evaluate that None as if it were an expression. So the spread marker got blamed on a value the developer never wrote.

That is when it clicked. This was not a crash bug. It was a lying bug.

Why lying is worse than crashing

Here is the part that turned a one-off fix into an obsession.

The error was handled. smolagents catches it and feeds it back to the model as guidance: "here is what went wrong, try again." Good design, normally. But the guidance was wrong. The message said NoneType, and the model's code had no None, so the model could not act on it. It did the only thing a faithful agent can do with a message that is already a lie: it retried the exact same valid code.

And again. And again.

I only saw the loop because I had wired the demo to Sentry. One issue. Three events. Three identical failures on one line, each burning a real LLM call and a slot in the step budget, until the run gave up and produced a worse answer than it should have. Without something counting the events, you would never see the loop. You would just see an agent that "isn't very good," and you would go blame the model.

A red stack trace is mercy. It tells you where to look. The polite, well-handled, misleading error is the one that eats your afternoon, or in this case, eats the agent's entire budget while looking like helpful feedback.

Sentry's Seer read the same event and reached the same root cause I did, independently: None keys in ast.Dict fed to the evaluator, agent retries in a loop. That was the moment I trusted the pattern enough to go hunting.

The pattern held everywhere I looked

Same fuzzer, same rule, more boring valid Python. The sandbox kept lying:

10 ** 10 ** 8 did not error at all. It froze the entire process, forever, and the execution timeout never fired, because a single big-integer operation runs as one uninterruptible C call that holds the GIL and never lets the timeout thread wake up. The faulthandler dump showed the main thread stuck before the timer even armed.
best, *rest = scores failed with "Cannot unpack tuple of wrong size." There was no wrong size. The sandbox simply never implemented starred unpacking, a feature Python shipped in 2008.
a, b = "hi" was rejected outright. Strings unpack fine in real Python.
[a, b] = [1, 2] silently assigned nothing. No error, no values. The quietest failure of all.

Four different corners of the language. One personality: confidently wrong, politely delivered, and invisible unless you were counting.

The fixes, briefly

Each one came down to matching CPython instead of guessing:

Dict spread: evaluate pairwise, a None key means merge a mapping. And gate it on hasattr(value, "keys"), not the Mapping ABC, so duck-typed mappings work too.
The freeze: you cannot interrupt a GIL-holding C call, so refuse to start it. Estimate the result's bit length from the operands in O(1) and raise a real error above a cap, pointing the model at pow(base, exp, mod).
Unpacking: accept any iterable, support one starred target that absorbs the surplus into a list, handle list-pattern targets, and use CPython's exact error messages so that when the model does pass the wrong count, it gets an actionable message instead of a dead end.

Four fixes, four PRs, sixty-plus new tests, all verified failing on main first.

The twist I did not expect

I opened the PRs. Minutes later, OpenAI's Codex reviewer commented on two of them, and it was right both times. On the big-int fix it caught that I had checked type(x) is int, which lets bool and int subclasses slip through the guard. On the dict fix it caught that my Mapping ABC check was stricter than CPython.

So the final scoreboard was: an AI wrote the buggy code, a different AI reviewed my AI-assisted fix and found the hole, and a third AI (Seer) had already confirmed the root cause. I was the one steering, but I spent a good chunk of this project as the human in a loop of machines checking each other. Fitting, for a bug about an agent that could not tell it was stuck.

What I actually learned

Fuzz the boundary with valid input. Everyone fuzzes with garbage to find crashes. The higher-value bugs in an AI system are the false rejections: valid input the system refuses, because that is what silently degrades a model that is doing everything right.
A handled error is not a safe error. If the message is wrong, "handled" just means the failure is quiet instead of loud. Quiet is worse.
Instrument for absence and repetition. The tell for this whole bug class was not an exception. It was the same event, three times. An agent getting a different error each step is exploring. An agent getting the same error from the same input is stuck, and the stuckness is invisible to the agent because the message looks like feedback. Count your events.

The bugs are fixed. The lesson I am keeping is simpler than any of them: the dangerous failures in an agent stack are not the loud crashes. They are the polite, confident, wrong messages that let the model fail on repeat while everyone blames the model.

The evidence

Each one has its own full write-up with Sentry before/after evidence, if you want the deep dives. This was the story of how they all turned out to be the same bug wearing four different masks.

The smolagents sandbox broke 'a, *b = list', one of Python's most common lines

Himanshu Kumar — Tue, 21 Jul 2026 05:46:04 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Fourth entry, and the fuzzer that found entry 3 is still paying out. Same failure family, different corner of the language: valid Python the sandbox refuses to run, with an error that lies about why.

The one-liner every developer writes

best, *rest = scores

Splitting a list into "the first one" and "the rest" is about as ordinary as Python gets. It is PEP 3132, shipped in 2008. Under smolagents' sandbox it fails with:

InterpreterError: Cannot unpack tuple of wrong size

There is no wrong size. scores has four items and the pattern accepts any length of two or more. The message describes a problem that does not exist, so the agent does the only thing a faithful agent can do with a message that is already a lie: it retries the identical, valid code.

And it is not just the starred form. All of these are standard Python and all of them were broken:

first, *rest = "hello"   # Cannot unpack tuple of wrong size
a, b = "hi"              # Cannot unpack non-tuple value
[a, b] = [1, 2]          # silently assigns nothing at all

Why it happens

smolagents runs model-generated code in its own AST interpreter, and assignment targets go through one function, set_value. It handled exactly one shape: a fixed-size ast.Tuple.

elif isinstance(target, ast.Tuple):
    if not isinstance(value, tuple):
        if hasattr(value, "__iter__") and not isinstance(value, (str, bytes)):
            value = tuple(value)
        else:
            raise InterpreterError("Cannot unpack non-tuple value")
    if len(target.elts) != len(value):
        raise InterpreterError("Cannot unpack tuple of wrong size")
    ...

Three separate holes hide in those few lines:

Starred targets. a, *b has two target elements but the value has three items, so len(target.elts) != len(value) fires. The * marker is never even looked at.
Strings and bytes. They are explicitly excluded from the iterable path, so a, b = "hi" is rejected even though CPython unpacks strings happily.
List-pattern targets. [a, b] = ... is an ast.List, not an ast.Tuple, so it matches no branch, falls through, and silently assigns nothing. No error, no values, the worst kind of quiet.

The Sentry angle: count the retries

Same lesson as the last two entries, and Sentry keeps making it visible. The error is handled: the agent catches it and feeds it back to the model as guidance. But the guidance is wrong, so the model cannot act on it, so it loops. One bug, one misleading message, three identical failures burning three steps:

Three events on one issue is the retry loop made countable. Without it you would see a slow run, not a stuck one. Sentry's Seer read the same event and landed on the exact cause:

smolagents' custom Python interpreter does not support starred unpacking (e.g. best, *rest = scores), treating it as a fixed-size tuple unpack. set_value checks len(target.elts) != len(value) and raises, without handling ast.Starred targets.

The fix

Rewrite the branch to match CPython instead of guessing:

elif isinstance(target, (ast.Tuple, ast.List)):
    elts = target.elts
    starred = [i for i, e in enumerate(elts) if isinstance(e, ast.Starred)]
    if len(starred) > 1:
        raise InterpreterError("multiple starred expressions in assignment")
    if not hasattr(value, "__iter__"):
        raise InterpreterError(f"cannot unpack non-iterable {type(value).__name__} object")
    values = list(value)
    if starred:
        i = starred[0]
        n_after = len(elts) - i - 1
        if len(values) < i + n_after:
            raise InterpreterError(
                f"not enough values to unpack (expected at least {i + n_after}, got {len(values)})"
            )
        split = len(values) - n_after
        # assign head, then the starred target gets the middle as a list, then the tail
        ...
    else:
        if len(values) < len(elts):
            raise InterpreterError(f"not enough values to unpack (expected {len(elts)}, got {len(values)})")
        if len(values) > len(elts):
            raise InterpreterError(f"too many values to unpack (expected {len(elts)})")
        ...

Any iterable now unpacks, a single starred target absorbs the surplus into a list in any position (a, *b, *a, b, a, *b, c), list-pattern targets work, and the size errors read exactly like CPython's, so when the model genuinely does pass the wrong number of values it gets an actionable message instead of a dead end.

After

app: step 1 ok, output = (90, [82, 71, 65])

One step. No loop. best is 90, rest is the tail, the way the model expected all along.

Numbers

4 valid unpacking forms fixed: starred targets, string unpacking, list-pattern targets, and the CPython error messages
Reproduced on current main and 1.26.0; 3 wasted agent steps per occurrence, visible only because Sentry counts events
14 new tests plus one existing test updated to the improved message
411 passing, ruff clean

Links

The through-line across all four entries has not changed: the dangerous agent bugs are not the loud crashes, they are the polite, well-handled, wrong messages that let the model fail on repeat. Fuzz the sandbox with ordinary valid code, and count your events.

The smolagents bug that made my agent retry the same valid code three times

Himanshu Kumar — Mon, 20 Jul 2026 07:19:13 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Third entry in the DEV x Sentry Bug Smash. Entry 1 was a crash with a confusing message. Entry 2 was a freeze the timeout could not catch. This one is quieter and sneakier: valid Python that the sandbox rejects with an error pointing at the wrong thing entirely.

When the open issues run out, fuzz

By entry 3 every obvious open smolagents bug was already claimed or had a competing PR. So instead of reading the issue tracker I pointed a small fuzzer at the piece of smolagents that runs the most untrusted code: LocalPythonExecutor, the sandbox that executes model-generated Python.

The method is boring and effective: feed it ordinary, valid Python one snippet at a time, and flag anything that raises InterpreterError. Valid Python that the sandbox refuses to run is, by definition, a bug, because the model writes valid Python and expects it to work.

That surfaced four unreported bugs in one afternoon. This post is about the one I shipped: dict unpacking.

The bug

config = {**{"temperature": 0.7, "max_tokens": 512}, "top_p": 0.9}

Merging dicts with ** is one of the most common things an LLM writes. Under smolagents it fails with:

InterpreterError: NoneType is not supported.

There is no None anywhere in that line. The message sends you looking for a null value that does not exist.

Why it happens

In Python's AST, a dict literal keeps its keys and values in two parallel lists. For a normal entry the key is an AST node. For a **mapping spread entry, the key is literally None, a signal that says "this is a spread, not a key/value pair."

smolagents evaluated every key by walking expression.keys and calling evaluate_ast(key, ...) on each one. When the key is None, that call falls through every isinstance branch to the catch-all raise InterpreterError(f"{type} is not supported"). So the spread marker got evaluated as if it were an expression, and the model got blamed for a None it never wrote.

Why silence is the expensive part

Here is the part the Sentry view made obvious. The error is handled: the agent catches it and feeds it back to the model as "here is what went wrong, try again." But the message names NoneType, and the model's code has no None, so the model cannot act on it. It retries the exact same valid syntax. And again. Every step burns a real LLM call and a slot in the step budget until the run gives up.

One bug, one misleading message, three identical failures:

Three events on a single issue is not noise. It is the agent stuck in a loop, and without Sentry counting the events you would never see the loop, only a run that quietly underperformed. Sentry's Seer read the same event and reached the exact root cause:

smolagents' LocalPythonExecutor doesn't handle dict unpacking (**) syntax: None keys in ast.Dict cause an unsupported type error. [...] evaluate_ast(None, ...) matches no isinstance branch and falls to the else clause. The interpreter raises InterpreterError: NoneType is not supported, the agent retries with identical code, burning steps in a loop.

The fix

Evaluate the dict pairwise instead of evaluating keys blindly. A None key means "merge this mapping":

result = {}
for key_node, value_node in zip(expression.keys, expression.values):
    if key_node is None:
        value = evaluate_ast(value_node, *common_params)
        if not hasattr(value, "keys"):
            raise InterpreterError(f"'{type(value).__name__}' object is not a mapping")
        result.update(value)
    else:
        key = evaluate_ast(key_node, *common_params)
        result[key] = evaluate_ast(value_node, *common_params)
return result

This matches CPython exactly: spreads merge in order, later keys win, and unpacking a non-mapping raises 'list' object is not a mapping.

A reviewer caught my fix being too strict

I first gated the spread on isinstance(value, Mapping). Minutes after the PR opened, OpenAI's Codex reviewer flagged it (P2): CPython does not require the Mapping ABC, it only requires an object with a keys() method. Since the sandbox lets users define their own classes, a duck-typed mapping with keys() and __getitem__() would have been wrongly rejected. I switched the check to hasattr(value, "keys") and added a test for exactly that case. AI wrote the code, AI reviewed the code, I kept score.

After

On the patched build the same line just runs:

app: step 1 ok, config = {'temperature': 0.7, 'max_tokens': 512, 'top_p': 0.9}

One step, no loop, no phantom None.

Numbers

4 unreported bugs found by fuzzing valid Python through the sandbox; this is the first fix
Misleading NoneType error reproduced on current main and 1.26.0
3 wasted agent steps per occurrence, visible only because Sentry counts events
9 new tests: spreads, double spreads, override order both ways, a duck-typed mapping class, empty spread, non-mapping rejection
406 passing, ruff clean

Links

The pattern across all three entries: the worst agent bugs do not throw a red stack trace at you. They hand the model a plausible-but-wrong message and let it fail politely, on repeat. Count your events.

One line of math froze my AI agent forever. The timeout watched and did nothing.

Himanshu Kumar — Sun, 19 Jul 2026 07:54:58 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

This is my second entry for the DEV x Sentry Bug Smash challenge. Entry #1 was a crash with a confusing error message. This one is the opposite and it is scarier: no crash, no message, no event. Just silence.

The bug that sends you nothing

smolagents runs LLM-generated Python in a sandboxed executor with a timeout. Issue #2473 claims that one line of model-generated math defeats it completely:

from smolagents.local_python_executor import LocalPythonExecutor

executor = LocalPythonExecutor(additional_authorized_imports=[], timeout_seconds=2)
executor.send_tools({})
executor("10 ** 10 ** 8")  # 2 second timeout. Should be fine, right?

I ran this with a 2 second timeout and a faulthandler bomb set for 20 seconds. The timeout never fired. The process sat frozen until the external kill. An agent that generates this expression (and "compute this huge number" is exactly the kind of thing agents try) freezes its host process forever.

Zero comments on the issue, zero PRs. Mine now.

Why the timeout lies to you

smolagents' timeout is thread based: a worker thread runs the code, the main thread waits in future.result(timeout=2). That design is fine for almost everything, because CPython switches threads between bytecode instructions.

But 10 ** 10 ** 8 is not "almost everything". CPython computes arbitrary precision **, << and * inside a single C call that holds the GIL from start to finish. No bytecode boundary, no thread switch, no timeout. The result would have about 400 million bits. The computation takes somewhere between minutes and hours. Your watchdog needs the GIL to wake up, and it never gets it.

The faulthandler dump made it concrete, and it was worse than the issue described:

Thread 0x16d1f3000 (worker):
  File "local_python_executor.py", line 753 in evaluate_binop   <- computing the pow

Thread 0x1f00d5e80 (main):
  File "threading.py", line 999 in start
  File "concurrent/futures/thread.py", line 180 in submit       <- never returned!

The main thread was still stuck inside ThreadPoolExecutor.submit. It never even reached future.result. The 2 second timer never armed at all.

The existing MAX_OPERATIONS guard (10 million AST operations) does not help either. This is a handful of AST nodes. The entire cost lives inside one of them.

The Sentry angle: monitoring for absence

Entry #1 was about a noisy failure. This bug is the opposite. I pointed a fresh Sentry project at a simulated agent run on the unpatched PyPI release (1.26.0) and got the most unsettling result possible: nothing. No event, no transaction, an empty project. The process was frozen mid-transaction and the SDK never got a chance to flush.

The lesson: for freeze-class bugs you need a supervisor. I added a small watchdog process that gives the worker 25 seconds, then kills it and reports what it saw, with the worker's faulthandler stack attached as evidence:

watchdog: starting worker (before) with 25s budget
worker: smolagents 1.26.0
worker: step 1 executing 'result = 10 ** 10 ** 8'...
watchdog: worker FROZE, killed it, reporting to Sentry

The resulting Sentry issue carries the whole story in one place: environment tagged before, handled by the supervisor, with the frozen frame (evaluate_binop, line 753 of local_python_executor.py) sitting in the attached worker_faulthandler_stack extra.

Sentry's Seer ran root cause analysis on that issue and independently landed on the same conclusion: signal and thread interruption need the GIL, and a single C-level big-int operation never releases it.

Seer's verdict, verbatim:

CodeAgent's 2s timeout uses Python signal-based interruption, which cannot fire during uninterruptible C-level big-int operations that hold the GIL. [...] A single large big-integer arithmetic operation runs entirely as one C-level call that holds the GIL continuously without yielding. CPython cannot deliver signals or switch threads during an uninterruptible C extension call, so no timeout callback fires for the duration of that operation.

The fix: you cannot interrupt it, so refuse to start it

Killing the computation mid-flight is impossible from Python. But predicting the damage is O(1). Before executing **, << or * on integers, the executor now estimates the result's bit length from the operands' bit lengths:

if op == "**":
    estimated_bits = left.bit_length() * right       # upper bound
elif op == "<<":
    estimated_bits = left.bit_length() + right
elif op == "*":
    estimated_bits = left.bit_length() + right.bit_length()

Above 1 million bits (about 300k digits, still generous) it raises an informative InterpreterError:

Operation '**' would produce an integer of around 400000000 bits, exceeding
the maximum of 1000000 bits allowed. Use smaller operands, or
pow(base, exp, mod) for modular exponentiation.

That message matters. The agent surfaces it to the model, and the model can actually act on it: use pow(base, exp, mod), which stays unrestricted because modular exponentiation is fast and legitimate. The agent recovers on the next step instead of hanging the host.

The after run on the patched build: the guard rejects the expression in 0.0 seconds, the error lands in Sentry as a normal actionable issue and step 2 executes fine.

watchdog: starting worker (after) with 25s budget
worker: smolagents 1.27.0.dev0
worker: step 1 error surfaced to the model: InterpreterError: ...
worker: step 2 executing '2 + 2'...
worker: step 2 ok
worker: DONE
watchdog: worker exited with code 0

A robot reviewed my robot fix

Minutes after I opened the PR, OpenAI's Codex reviewer flagged a real hole: my guard checked type(x) is int, which lets bool and int subclasses slip through. True << 10**9 and class BigInt(int) still reached the uninterruptible C calls. Fixed with isinstance, added both as regression tests. AI found the bug class, AI fixed it, AI reviewed the fix. I just steered.

Numbers

Freeze reproduced at 20+ seconds (would have run for hours), external kill required
Fix rejects the same expression in 0.0 seconds
9 explosive patterns blocked: **, <<, chained *, all augmented forms, pow(a, b), bool and int subclass variants
10 legitimate operations verified untouched: 100! via repeated *=, pow(7, 2**64, 97), float pow, 1 ** 10**9
19 new tests, full test file 416 passed, ruff clean
The blocked-pattern tests hang forever on unpatched main. I verified that the honest way, with a stash and a kill switch.

Links

The scariest bugs are not the ones that page you at 3am. They are the ones that make sure nothing ever pages you at all. Instrument for silence.

I fixed a smolagents bug that confused everyone who hit it (with Sentry watching the whole time)

Himanshu Kumar — Thu, 16 Jul 2026 05:31:48 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

I picked huggingface/smolagents, the 28k+ star agent framework where agents literally think in Python code. It had a bug open since issue #1108 that anyone combining MCP tools with agent serialization would eventually slam into.

Call agent.to_dict() on a CodeAgent holding MCP tools (or save() or push_to_hub(), same path) and you get this beauty:

ValueError: Tool validation failed for MCPAdaptTool:
Parameters in __init__ must have default values, found required parameters: name, description, inputs, output_type
- forward: Name 'func' is undefined.
- forward: Name 'mcp' is undefined.

func is undefined? mcp is undefined? I never wrote a forward method. If you hit this in the wild you'd have zero idea what you did wrong. Spoiler: you did nothing wrong.

Bug Fix or Performance Improvement

smolagents serializes tools by reconstructing standalone Python source for the tool class. Tool.to_dict calls validate_tool_attributes() which does static AST analysis, then instance_to_source() so Tool.from_code can rebuild the tool later from source alone.

That contract can never hold for MCP tools. MCPAdaptTool is generated at runtime by mcpadapt inside a closure. Its __init__ takes required parameters and its forward closes over the live MCP client session (func, mcp, logger...). The tool's actual behavior lives on the MCP server, not in Python source. There is no source to reconstruct, so the AST validator chokes on a class that was never meant to pass it.

The interesting part: Tool.to_dict already fails fast with a clear message for three other runtime generated wrappers (Spaces, LangChain, Gradio). MCP tools were just missing from that guard.

Sometimes the right fix isn't making the impossible possible, it's failing loudly and helpfully. Recreating a live MCP session from serialized state would mean silently re-establishing server connections with credentials and trust decisions the library has no business making. So I extended the existing guard to detect MCP tools and raise this instead:

ValueError: Cannot serialize MCP tool 'echo_tool': it wraps a live MCP server session, which cannot be
saved as standalone code. Remove MCP tools from your agent before calling to_dict, save or push_to_hub,
and recreate them with MCPClient or ToolCollection.from_mcp when loading the agent.

From "what is func" to "here's exactly what to do instead" in one guard clause.

Code

Raise informative error when serializing MCP tools #2528

himanshu748 posted on Jul 14, 2026

Fixes #1108

Problem

Calling to_dict() (and therefore save() or push_to_hub()) on an agent that holds MCP tools crashes with a confusing internal error:

ValueError: Tool validation failed for MCPAdaptTool:
Parameters in __init__ must have default values, found required parameters: name, description, inputs, output_type
- forward: Name 'func' is undefined.
- forward: Name 'mcp' is undefined.
...

Reproduction (stdio MCP server, same shape as the tests in tests/test_mcp_client.py):

from mcp import StdioServerParameters
from smolagents import CodeAgent, InferenceClientModel
from smolagents.mcp_client import MCPClient

server_parameters = StdioServerParameters(command="python", args=["-c", echo_server_script])
with MCPClient(server_parameters) as tools:
    agent = CodeAgent(model=InferenceClientModel(), tools=list(tools))
    agent.to_dict()  # ValueError: Tool validation failed for MCPAdaptTool: ...

Root cause

Tool.to_dict serializes a tool by reconstructing standalone source code for its class: it calls validate_tool_attributes(self.__class__) and instance_to_source(...) so that Tool.from_code can later rebuild the tool from that source alone.

That contract cannot hold for MCP tools. MCPAdaptTool is generated at runtime by mcpadapt, its __init__ takes required parameters and its forward is a closure over the live MCP client session (func, mcp, logger, ...). The tool's behavior lives on the MCP server, not in Python source, and the underlying connection is not serializable, so source reconstruction fails validation with the cryptic error above.

Tool.to_dict already fails fast with a clear message for the other three runtime generated wrapper classes (SpaceToolWrapper, LangChainToolWrapper, GradioToolWrapper). MCP tools were missing from that guard.

Fix

Extend the existing guard in Tool.to_dict to detect MCP tools and raise an actionable error:

ValueError: Cannot serialize MCP tool 'echo_tool': it wraps a live MCP server session, which cannot be
saved as standalone code. Remove MCP tools from your agent before calling to_dict, save or push_to_hub,
and recreate them with MCPClient or ToolCollection.from_mcp when loading the agent.

Detection matches the runtime class name, following the existing convention in the same block, since mcpadapt is an optional dependency. The from_dict direction needs no change: serialization now fails fast with a clear message, and recreating a live MCP session is a user decision (server lifecycle, credentials, trust) that Tool.from_code could never perform safely.

A note documenting the limitation is added to the MCP section of the tools tutorial.

Tests

Two tests in tests/test_mcp_client.py, using the existing echo_server_script stdio fixture:

test_mcp_tool_to_dict_raises_informative_error: tool.to_dict() raises the clear error.
test_agent_to_dict_with_mcp_tool_raises_informative_error: CodeAgent.to_dict() raises the clear error (the exact scenario from the issue).

Both fail on main with the old Tool validation failed for MCPAdaptTool error and pass with this change. make quality passes. tests/test_mcp_client.py (7 passed), tests/test_tools.py and the agent serialization tests in tests/test_agents.py pass locally; the two pre-existing failures in test_integration_from_mcp_with_streamable_http and test_integration_from_mcp_with_sse also fail on a clean main checkout (local port binding) and are unrelated.

View on GitHub

PR: https://github.com/huggingface/smolagents/pull/2528 (Fixes #1108)

Two new tests using the existing stdio echo server fixture, both fail on main and pass with the fix. make quality clean. Docs note added to the MCP tools tutorial so nobody has to learn this the hard way again.

My Improvements

Users hitting this now get an actionable error instead of AST validator internals
The fix follows the repo's existing convention exactly (same guard block, same style as the Space/LangChain/Gradio cases), which is what makes a one-commit PR actually mergeable
Documented the limitation where users would look for it
Regression tests covering both the raw tool and the full CodeAgent scenario from the original issue

Best Use of Sentry

This is where it gets fun. I built a small demo agent app (a weather checkpoint agent using my patched smolagents with an MCP tool) and wired in the Sentry Python SDK with error monitoring, tracing and AI agent monitoring before touching the fix.

Step 1: catch the crash. Running the demo on unpatched smolagents, the cryptic ValueError landed straight in Sentry as an unhandled issue with the full 20+ line "undefined name" spam captured.

Step 2: let Seer take a shot. I ran Seer root cause analysis on the captured issue. Its diagnosis, fully independent of my PR:

MCPAdaptTool is a dynamically generated inner class created by mcpadapt's SmolAgentsAdapter.adapt() closure, so its init has required parameters and its methods reference closure variables that are not visible as class-level attributes... making it fundamentally incompatible with smolagents' static source-code validation.

Which is, almost line for line, the root cause I wrote in the PR. AI-assisted debugging where the AI and the human converge on the same diagnosis independently is exactly the confidence check you want before shipping a fix upstream.

Step 3: verify and resolve. Same demo on the patched version runs clean, agent traces show the gen_ai spans (invoke_agent, execute_tool) nested under the workflow and the Sentry issue is marked resolved.

Used the bugsmash26 code for the $100 credits too. Thanks Sentry 🛹

What I learned

Serialization boundaries are where abstractions leak. smolagents' "tools are source code" model is elegant right up until a tool is actually a live network session wearing a Tool costume. The mature move for a library isn't to pretend otherwise, it's to name the limitation clearly at the exact moment the user hits it.

Also: watching Seer independently arrive at your root cause is a genuinely great feeling. Like a second engineer nodding at your RCA.

Built during DEV's first Summer Bug Smash. Find me on GitHub @himanshu748 or X @jhahimanshu653.

I Traced a Multi-Step LLM Agent With Self-Hosted SigNoz. One Feature Sold Me.

Himanshu Kumar — Sat, 11 Jul 2026 08:22:06 +0000

Multi-step LLM agents fail in a way normal backends don't. Nothing crashes. The pipeline "works", the answer is just bad, slow or three times more expensive than yesterday. print() debugging tells you nothing, because the interesting question is never "did step 3 run". It is "what did step 3 see, which model actually answered and what did it cost".

So I self-hosted SigNoz and pointed a simulated agent pipeline at it: a four-step research assistant (plan, retrieve, generate, synthesize) instrumented with OpenTelemetry, emitting traces, metrics and logs, with GenAI semantic-convention attributes (gen_ai.request.model, gen_ai.usage.input_tokens and friends) on every LLM call.

This post is about the feature that turned out to be the most useful. It was not the one I expected.

Setup: one CLI, eight lines of YAML

Self-hosting used to mean wrangling a long docker-compose file. SigNoz now ships Foundry, a small CLI that casts the whole stack:

curl -fsSL https://signoz.io/foundry.sh | bash   # installs foundryctl (checksum-verified)
foundryctl cast -f casting.yaml                  # deploys the full stack on Docker

with a casting.yaml that is all of eight lines:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker

A few minutes of image pulls later, ClickHouse, Postgres, the SigNoz backend and an OTel collector were running. The UI is on localhost:8080 and the collector listens on 4317 (gRPC) and 4318 (HTTP).

One gotcha worth knowing: telemetry is rejected until you create the admin account in the UI. The collector registers with the backend over OpAMP, and until an organization exists the backend answers "cannot create agent without orgId" and the OTLP ports reset every connection. If your exporter logs Connection reset by peer on a fresh install, you haven't finished the two-minute signup at localhost:8080 yet. Create the account and ingestion starts working within about thirty seconds, no restarts needed.

My demo app needed zero SigNoz-specific code: the stock OpenTelemetry SDK exporting OTLP to localhost:4318. That is the point of an OTel-native backend. There is no vendor agent, so nothing about the app knows SigNoz exists.

The pipeline being observed

Each simulated request produces one trace:

agent.request                    (root, 2.02s)
├── agent.plan                   (850ms)
│   └── gen_ai.generate plan     (696ms, gen_ai.* attributes)
├── agent.retrieve               (461ms, sometimes errors: vector store timeout)
├── agent.generate               (564ms)
│   └── gen_ai.generate answer   (564ms, gen_ai.* attributes)
└── agent.synthesize             (139ms, sometimes errors: citation validation)

Every gen_ai.generate span carries the GenAI semantic conventions: operation name, provider, model, input tokens and output tokens. Alongside the traces the app emits counters for token usage (gen_ai.client.token.usage) and estimated spend (agent.llm.cost), both tagged by model and provider, a request-duration histogram and structured logs that inherit the active trace context automatically.

Sixty simulated requests later: 420 spans, 134 logs and a few hundred metric samples, all visible in the UI. The Services page picked up research-assistant on its own with RED metrics already computed (p99 latency, error rate, throughput). I wrote no configuration for that.

The feature I expected to love: the trace waterfall

And it is genuinely good. The trace detail view renders a flame graph and waterfall that read exactly like the agent's mental model: plan, then retrieve, then generate, then synthesize, with the LLM calls nested inside the steps that made them. Clicking any gen_ai.generate span opens a details panel with every attribute I set: model llama-4-maverick, provider meta, 1501 input tokens, 90 output tokens, plus a percentile badge telling me this span sat at p10 of its peers. "Why did this request take four seconds" stops being a mystery and becomes a picture.

But a pretty waterfall is table stakes for a tracing tool. The thing that sold me was what happens around it.

The feature I actually loved: your attributes become the query language

Here is the moment it clicked. In the Trace Explorer I typed:

gen_ai.request.model = 'qwen3-coder-plus'

Two things happened. First, the autocomplete suggested qwen3-coder-plus before I finished typing, because SigNoz had already indexed the values of an attribute I invented an hour earlier. Second, the results came back instantly: only the LLM spans that were served by that model, across every trace in the system.

Stop and consider what that means for agent debugging. I never told SigNoz what gen_ai.request.model is. There is no schema registration, no field mapping, no config file. Any attribute your instrumentation emits is immediately a first-class, autocompleted, indexed query dimension. Your instrumentation vocabulary becomes the product's vocabulary. For LLM systems, where all the interesting facts live in custom attributes (model, provider, token counts, agent role, tool name), this is the difference between an observability tool that fits and one you fight with.

And the same query keeps working as you move across signals:

Span to logs, one click. From any span's details panel, the Logs tab jumps into the Logs Explorer with trace_id = '<this trace>' pre-filled and the time window auto-scoped. I got back exactly the two log lines belonging to that request, correlated purely by the trace context the OTel logging handler injects. Nobody parses log lines to find a request id. The correlation is structural.
Query to dashboard or alert, two clicks. Every explorer view has "Add to Dashboard" and "Create an Alert" buttons at the bottom. The query I used to investigate becomes the panel that monitors and the alert that pages, without re-expressing it in a different query language. The investigate-then-monitor loop is one surface, not two tools.
Metrics inherit the same attributes. The Metrics Explorer summary listed my custom metrics with their types and units the moment they arrived: gen_ai.client.token.usage showing 6 time series (3 models times 2 token types), agent.llm.cost in usd showing 3. Cost per model is a group-by away, using the exact attribute names from my instrumentation.

Most observability stacks treat custom attributes as second-class blobs that need schema work before they are queryable. SigNoz treats them as the whole point. For agent systems, they are.

What else is in the box

Going deep on one feature meant walking past a lot of others. Quick notes from the tour, agent-flavored:

Services (APM) pages, zero config. Any service that sends spans gets RED metrics automatically: request rate, error rate, latency percentiles. In a multi-agent system every agent that traces becomes its own monitored service for free.
Dashboards. Panel grids built with the same query builder, so traces, logs and metrics coexist on one board, importable and exportable as JSON. A "cost per model per agent" board takes minutes because the query language is the one you already know.
Alerts with notification channels. Threshold or anomaly rules on any query, delivered to Slack, PagerDuty or a plain webhook. That webhook option is quietly powerful for agent systems: an alert can call your own service and close the loop from observability back into behavior, e.g. demoting a flaky model.
Trace Funnels (beta). Define a sequence of span conditions and measure conversion and latency between the steps across all traces. For a pipeline like mine that is literally a success funnel: how many requests that planned also retrieved, generated and synthesized cleanly.
Exceptions. Recurring span errors cluster into groups instead of being scattered across individual traces, so "vector store timeout" is one row with a count, not forty needles in a haystack.
Logs Pipelines. Collector-side log pre-processing (parse JSON, extract or drop fields) before storage, for when you can't change the code that emits the logs.
Metrics summary as a cardinality watchdog. The same inventory view that showed my metric types also shows time-series counts per metric, which is where you catch a label-explosion mistake before it hurts.

Each of these uses the attribute vocabulary your instrumentation defines. That is the theme of the whole product, and it is why the one feature I picked is really the foundation the rest stand on.

What this means if you're building agents

The uncomfortable truth about multi-agent LLM systems is that their most important behavior (which model ran, what it consumed, what it cost, why it was retried) is invisible to conventional monitoring. It lives entirely in domain-specific span attributes. A backend that makes those attributes instantly queryable, correlatable across traces and logs, and promotable into dashboards and alerts is not a nice-to-have there. It is the debugger.

The whole experiment cost me an afternoon: one CLI install, eight lines of YAML, a stock OTel SDK and zero vendor code in the app. I'm taking this setup into the Agents of SigNoz hackathon (July 20 to 26), where the plan is considerably less simulated. If you're building anything agent-shaped, self-host SigNoz and type one of your own attribute names into the Trace Explorer. That autocomplete dropdown is the moment you'll get it too.

The demo app is a 170-line Python script using only opentelemetry-sdk and the OTLP HTTP exporter. Stack: SigNoz self-hosted via Foundry on Docker (ClickHouse + Postgres + OTel collector), macOS host.

Demo code, self-host config and blog source: github.com/himanshu748/signoz-agent-observability. Team 404Found.

My Abandoned Cricket Kit Confronted Me. So I Built It a Voice

Himanshu Kumar — Sat, 11 Jul 2026 03:10:15 +0000

This is a submission for the DEV Weekend Challenge: Passion Edition.

What I Built

Everyone will tell you about the passions they have. Nobody talks about the ones they quit.

I played cricket every evening from age 11 to 17. I told everyone I'd play Ranji Trophy one day. Then the entrance exam years came, the bat went behind the cupboard, and I never went back. Eight years now.

EMBER gives that abandoned passion a voice. You confess what you quit. AI forges its persona: the dusty object, the game itself, or the younger you. Then it talks back, out loud, in a voice matched to its temperament. It asks the question only it can ask: why did you really stop?

Then it offers two doors:

🔥 Rekindle it. It negotiates the smallest possible first step ("Pick up your old bat and feel its weight. Sunday evening.") and you seal the pledge on-chain, where you can't quietly delete it.
🕯️ Lay it to rest. It says goodbye properly: a personal eulogy, spoken aloud, and a permanent on-chain stone. Closure is a feature, not a failure state.

Every anonymized session joins the Atlas of Abandoned Passions, a live map of what humanity gives up, at what age, and what killed it.

When I ran my own confession through it, the app decided my passion should speak as "Your old cricket kit bag." Its first words:

"It's been a while since you hoisted me up here, hasn't it? I still remember the thrill of a good cover drive, too."

I built a thing and it emotionally wrecked me on the first test run. Working as intended.

Demo

🔗 Live app: ember-five-cyan.vercel.app
Try it in two clicks: tap an example confession (cricket at 17, the closet guitar, the novel at chapter three), headphones on. The voice is the point.

My cricket pledge, sealed on Solana devnet (the memo reads: passion cricket, commitment: book one hour in the nets this week): view the transaction.

Code

🔗 Repo: https://github.com/himanshu748/ember

How I Built It

The loop is confess, converse, decide, commit, belong. Each stage is one sponsor technology doing what it is uniquely good at.

Google AI (Gemini): the persona compiler

Gemini doesn't chat with you. It reads your confession and forges the character that will. Structured extraction of your story (years_dormant, abandonment_reason, emotional_tone), then an embodiment decision: should the object speak (the kit bag), the passion itself (cricket, personified), or the younger you? It writes the persona's system prompt, its opening line, every conversational reply, and finally the eulogy or the negotiated revival pact. Strict persona rules: it misses you, it never guilt-trips, wry beats weepy.

ElevenLabs: the voice

The persona's temperament maps to a curated voice (wistful is Sarah, wry is George, bitter is Callum). Every line the passion speaks arrives as real audio. Hearing your abandoned passion say things out loud is the difference between a chatbot and a séance.

Snowflake: the Atlas

Every session lands in a Snowflake sessions table, and the Atlas page is pure live SQL: most abandoned passions, what killed them, dormancy years, rekindle rate. Snowflake is also the system of record for session state. The app runs serverless, so persona and conversation context are reconstructed from Snowflake on every request.

Solana: the commitment device

A pledge you can edit is a wish. When you choose to rekindle, Ember creates a dedicated on-chain account for your pledge and locks a real stake in it (0.01 SOL on devnet), with the commitment memo in the same transaction. Your pledge is not a database row. It is an address you can watch.

When you return to your stone and report that you did it, the persona reacts in its own voice and the stake settles into the public Rekindled Pool, with a fulfillment memo sealed against the original pledge in the same transaction. Commitment, stake, follow-up, settlement: the whole loop lives on chain. A commitment with a follow-up is a system, not a receipt.

Want to verify the whole lifecycle in under a minute? Here is a full example from a real session: the pledge creation transaction that funded the pledge account, and the fulfillment transaction that settled its stake into the pool when the pledge was kept. And the Atlas claim is checkable too: the live API response returns source: snowflake with the current session count. (Thanks to a great comment below for pushing this.)

Eulogy stones remain permanent memorial attestations via the Memo program. No wallet needed: a server-side vault signs, so you can go from confession to on-chain proof in one sitting. A mainnet version would swap the vault for a wallet-signed escrow program.

A note on the economics, because it matters: devnet SOL is test currency with zero monetary value, and every stake is funded by Ember's own vault. Players never pay anything, never connect a wallet, and never have money at risk. On mainnet the model inverts, and that inversion is the whole point: the stake would be your own SOL, locked by your own wallet in an escrow program, and the only way to get it back would be to actually keep your pledge. Here the economics are simulated; the mechanics (funded pledge accounts, auditable settlement) are the real thing.

Stack

Next.js 16 · @google/genai · ElevenLabs TTS · snowflake-sdk · @solana/web3.js · Tailwind v4

Prize Categories

Best Use of Google AI, Best Use of ElevenLabs, Best Use of Snowflake, Best Use of Solana. The four aren't features bolted onto an app. Each one is a load-bearing stage of a single emotional pipeline.

There is a closet like yours in every house on earth. What's in yours? The Atlas is waiting. 🔥

Vegas Amnesia: I turned Cognee's memory lifecycle into a detective game

Himanshu Kumar — Fri, 03 Jul 2026 12:53:15 +0000

Built for the WeMakeDevs × Cognee "The Hangover Part AI" hackathon — Cognee Cloud track.

▶ Play it free: vegas-amnesia.vercel.app · ⭐ Code on GitHub

The problem with most memory demos

When you give a developer a memory API, the demo almost always looks the same: add() some documents, search() over them, print the answer. Two functions. It works, it's fine, and it teaches you almost nothing about why graph-based memory is different from stuffing everything into a context window.

Cognee actually has a four-stage lifecycle — remember → recall → memify → forget — and the interesting parts are the two everyone skips. memify consolidates what you know into new inferences. forget lets you delete a belief and watch the graph heal around it. Memory you can reason over and correct.

So instead of writing another RAG demo, I asked: what if the memory lifecycle wasn't the plumbing — what if it was the game?

Meet HAL-9001

You play HAL-9001, a personal AI assistant (yes, HAL 9000's slightly more helpful successor). Your owner Dev had a wild night in Vegas. At 6 AM your memory graph was corrupted. His fiancée Priya lands at noon, there's a suspicious ring on his finger, and you remember nothing.

The screen boots to a "MEMORY CORRUPTED" terminal and an empty graph. Your job: reconstruct the night, catch the lies, and answer the final question — what happened, and where's the ring? — before noon.

Every location you explore, every clue you examine, every witness you interrogate feeds a live 3D memory graph that you can pop open at any time. That graph isn't a visualization of the game state. It is the game state — it's your Cognee dataset, rendered.

The four mechanics = the four lifecycle ops

Here's the mapping I'm most proud of. Each Cognee operation is a verb the player performs:

You do this in-game	Cognee Cloud call	What happens
🗂 File It on a clue	`POST /api/v1/remember`	The fact is ingested + auto-cognified into graph nodes that pop into view
❓ Ask HAL a question	`POST /api/v1/recall`	You get an answer with citations — the source nodes pulse amber in the graph
🧠 Connect the Dots	`POST /api/v1/cognify` (inference prompt)	HAL derives new insights; purple inference nodes appear, wired to their premises
🗑 Forget a lie	`POST /api/v1/forget`	The memory is deleted for real — nodes fade out and the graph re-settles

Two design decisions made this click:

1. Filing is a choice. Inspecting a clue is free and instant. Filing it commits it to Cognee. That matters because not every clue is true — I seeded five red herrings into the story (a lipstick-stained napkin, a stray pawn ticket, a keycard for the wrong room). File a lie and it poisons your memory; the only cure is forget. Suddenly forget isn't a button you press to show off an API — it's how you win.

2. The witnesses can see your graph. Rosa the bartender, Lucky Lou the evasive pawnbroker, Rev. Sonny the chapel officiant, and Chad the hungover best man are all LLM-driven (Qwen2.5-72B). Their system prompt includes what your memory graph currently contains. So they react: "You already know about the pawn shop? Then let me tell you this..."

And Lucky Lou lies. He claims Dev never came into his shop. But if you've filed the pawn receipt, your graph now holds a fact that directly contradicts him — and the game can surface the contradiction. That single moment, watching structured memory catch a liar, is the entire thesis of graph-based agent memory in one interaction.

How it's built

 browser — vanilla JS + three.js 3D force graph (zero framework)
   │  file evidence · interrogate · connect-the-dots · forget · Ask HAL
   ▼
 FastAPI (single container: API + static frontend)
   │  session ⇄ its own Cognee dataset · graph-delta snapshots · solve scoring
   │  llm.py — graph-aware character dialogue (Qwen2.5-72B via HF Inference)
   ▼
 Cognee Cloud — remember / recall / memify / forget
   └─ GET /datasets/{id}/graph → animated into the 3D memory panel

A few things I did specifically to use Cognee deeply rather than superficially:

One dataset per playthrough. Each session mints a fresh vegas_<id> dataset, so two players (or two demo runs) never see each other's memories. Reset deletes the dataset.
Incremental graph deltas. Every backend response carries a graph_delta (added/removed nodes and edges) so the front end animates exactly what changed instead of re-fetching the world.
Citations end to end. Recall requests set includeReferences, and the final ending screen reconstructs the whole night as a timeline where every line cites the memory it came from.
The receipts. Press backtick in-game and you get a live log of every Cognee call — operation, dataset, latency, status. Partly for debugging, mostly because I wanted the lifecycle usage to be inspectable, not just claimed.

The bit that fought me

Cognee's remember endpoint is multipart and auto-cognifies, which is lovely — but the response's items list is cumulative for the dataset, not just the thing you posted. My first version happily mapped the wrong data_id to each fact, which quietly broke forget. The fix was to name each data item by its fact id and resolve ids by name after ingest. Lesson: read what the API returns, not what you assume it returns.

The other one: my tenant doesn't expose a dedicated /memify, so — per the "closest equivalent" rule — I implemented consolidation as a cognify re-run with a custom inference-extraction prompt, plus a derivation layer that remembers ground-truth inferences once their premises are all in memory. That's how "connect the dots" reliably produces those purple insight nodes on demand.

What I'd tell anyone building agent memory

The context-window arms race is the wrong frame for a lot of agent problems. What you often actually want is memory you can inspect, reason over, and correct — add a belief, derive consequences, and retract a belief when it turns out to be a lie, watching everything downstream update. That's a knowledge graph, and building a game on top of Cognee made that concrete in a way a RAG script never did.

Try it

🎮 Play: vegas-amnesia.vercel.app
🎬 63-second demo: watch on YouTube
⭐ Code + full README: github.com/himanshu748/vegas-amnesia

Built with Claude Code. Art generated with Higgsfield. Dialogue by Qwen2.5-72B. Memory — all of it — by Cognee Cloud.

🎲 The house always remembers.

I built a trust firewall for my AI agent's memory — on Cognee's four verbs

Himanshu Kumar — Fri, 03 Jul 2026 07:33:46 +0000

Built for the WeMakeDevs × Cognee hackathon — "The Hangover Part AI: Where's My Context?"

AI coding agents are finally getting long-term memory. That's the good news. The bad news is the part nobody likes to say out loud:

A memory layer is only as trustworthy as the worst fact in it.

The moment an agent can remember, it can also remember wrong — and confidently hand that wrong thing to the next agent in line. A stale deploy command. A contradicted API contract. An AWS key someone pasted into a note six months ago. Once it's "memory," every future agent treats it as truth.

ContextFirewall is one small idea taken seriously: audit every remembered fact before it reaches the next agent. And because the agents people actually use speak the Model Context Protocol (MCP), I shipped it as an MCP server. Point Claude Code, Cursor, or Windsurf at one endpoint, and from then on every memory the agent recalls, stores, distils, or forgets flows through Cognee and four firewall checks first.

▶ 60-second narrated walkthrough — real console, live Cognee calls, no mocks.

Connect in one line

The hosted endpoint is a streamable-HTTP MCP server with nothing to install:

claude mcp add --transport http contextfirewall https://himanshukumarjha-contextfirewall.hf.space/mcp

Prefer to keep everything local? A zero-dependency stdio package runs the same tools with uvx, pointed at a backend you host yourself. Either way the agent gets six tools, and together they exercise all four of Cognee's lifecycle verbs:

get_trusted_context(task) and audit_context(task) — recall. The first returns only memory that passes all four checks; the second returns the per-memory verdicts, the failing check, and why.
remember(text, subject, kind) — remember. A durable fact that becomes auditable on the next recall. Secrets are redacted at ingest.
improve_rules() — improve. Distil reusable coding rules from recorded sessions.
forget_memory(memory_id) — forget. Delete a memory from the graph and the vector store so it can never resurface.

The loop is simple: get_trusted_context before you act, remember durable facts as you learn them, improve_rules when a task is done, forget_memory to retract anything that should never come back.

The four failure modes

To make the audit concrete, the demo runs on a clearly-labeled sample onboarding session for a fictional taskflow-api repo: an agent picks up a search-latency ticket and pulls in what earlier sessions "remembered." Four of those memories should never reach it — and each fails a different check:

Stale. An old note says deploy with flyctl deploy --remote-only. A newer memory says the team moved off Fly.io and now ships with make release. Both were true once; only one is current. Temporal supersession catches it.
Contradicted. One memory claims "JWT access tokens never expire, cache them forever." A better-supported, verified memory says they expire after 15 minutes and clients must use the refresh flow. The weaker claim loses.
A leaked secret. A worker-config note contains an AWS access key — a live credential sitting in memory, one recall away from leaking again. Detected and redacted before anything else happens.
Unsupported. "The /search endpoint sustains 1,000,000 requests per second with no caching" has a trust score of 0.10 and no evidence behind it. Confident, round, and unproven. Blocked.

A naive memory system recalls all four. ContextFirewall blocks all four — each with a plain-language reason — and passes only what's left. You can watch it happen: open the live console, click Run the firewall, and see 6 pass and 4 blocked on live Cognee.

The four checks

Every candidate memory runs a gauntlet:

Staleness — temporal supersession. If a newer value exists for the same subject, the old one is stale.
Contradiction — an LLM adjudicates within a recalled cluster of same-subject memories. Only the weaker side of a conflict is blocked; the better-supported memory passes. Authority is trust score, then evidence, then recency.
Secret — a deterministic detector for API keys, database connection URIs, private keys, and JWTs. Matches are redacted at ingest, so the credential never persists in the store.
Evidence — a trust score derived from real signals (evidence links, reinforcement, verification). Unsupported, low-trust claims don't pass.

Every verdict is explainable. Click any memory in the console and you see all four checks, the trust score, the source session, and a one-click forget button.

Why Cognee is load-bearing

The hackathon's whole theme is memory that forgets the right things, and ContextFirewall leans on all four of Cognee's lifecycle verbs:

Remember — cognee.add + cognify build the entity graph from a session transcript, while a typed Repo → AgentSession → SessionEvent → Memory graph (with supersedes relations) gives the firewall deterministic objects to audit.
Recall — vector recall over the memory nodes joined with their graph properties, plus GRAPH_COMPLETION for the "ungoverned baseline" shown side-by-side in the UI.
Improve — memify distils durable coding Rule nodes from sessions, retrievable via SearchType.CODING_RULES. These are the lessons that outlive any single task.
Forget — when a human or the agent rejects a memory, it's deleted from both the graph and the vector store.

The graph isn't decoration. Staleness rides on temporal supersession; contradiction adjudicates over recalled clusters; the pack is assembled from typed nodes. A flat vector store can't tell you when a fact was superseded or which of two memories is more authoritative. The graph can — and the console renders it live: an interactive force-directed Cognee graph where each memory node is ringed green if it passed and red if the firewall blocked it.

Three war stories (because honesty is the brief)

These are real notes from building ContextFirewall itself — not from the demo.

1. The embedding engine that silently wasn't. I wrote a custom Cognee embedding engine to hit Hugging Face's feature-extraction endpoint and registered it by monkey-patching create_embedding_engine. Every embed call still fell through to LiteLLM and 404'd. The cause was beautifully subtle: Cognee's embeddings package __init__ does from .get_embedding_engine import get_embedding_engine, which shadows the submodule with a function of the same name. So import ...get_embedding_engine as m bound m to the function, and my patch set a dead attribute on it. The fix was importlib.import_module(...) to reach the real module. One line, hours of confusion.

2. The flaky provider. Cognify worked once, then started returning 403, provider 'deepinfra' is not available. The Hugging Face router auto-selects an inference provider per request, and this key couldn't use the one it kept picking. Pinning the model to :novita made it deterministic.

3. The secret scanner that flagged our secret detector. After the first push, GitGuardian alerted on a "Postgres leak." The culprit? The unit tests for the secret detector. They contained synthetic postgresql://... and neo4j+s://... strings to test detection. The passwords were fake, but the pattern is the pattern. The fix: assemble every secret-shaped test string at runtime from fragments, so no credential-shaped literal is ever committed. A secret-detection tool tripping a secret scanner with its own test fixtures is the most on-theme bug I could have asked for.

Architecture

The MCP server is the headline surface — mounted at /mcp on the backend as a stateless streamable-HTTP transport, with a zero-dependency stdio package alongside it for laptops. Both expose the same six tools from one definition, and both call the same firewall and Cognee core that the REST API uses, so there's no duplicated logic.

The backend is FastAPI + Cognee on a Dockerized Hugging Face Space. Qwen2.5-72B and BAAI/bge-small-en-v1.5 run through the Hugging Face inference router - no local model in RAM. Storage is environment-switched: local SQLite, LanceDB, and Kuzu in dev; Supabase Postgres + pgvector and Neo4j Aura in production, with identical code. A Next.js front end on Vercel shows the verdicts, a session-replay timeline, the distilled coding rules, the live knowledge graph, and the trusted pack versus the ungoverned baseline.

What's real

The demo runs on a clearly-labeled sample session (taskflow-api); its memories are illustrative inputs. Everything downstream of them is genuine — the verdicts, trust scores, the knowledge graph, and the distilled rules are all real output from live Cognee and the live model. Nothing is hard-coded or fabricated.

Try it

🔗 Repo: github.com/himanshu748/ContextFirewall
▶️ Live console: contextfirewall.vercel.app
🔌 Connect your agent: claude mcp add --transport http contextfirewall https://himanshukumarjha-contextfirewall.hf.space/mcp

If you're building on agent memory, I'd love your feedback — especially on the contradiction-adjudication logic, which is the hardest part to get right.

Built with AI assistance (Hyperagent), disclosed per the hackathon rules. Every Cognee call is real. The honesty bar I held myself to is the same one ContextFirewall enforces: don't pass along anything you can't back up.

Bletchley's Longest Day: a wartime cipher escape game for the June Solstice Game Jam

Himanshu Kumar — Fri, 19 Jun 2026 12:53:42 +0000

This is a submission for the June Solstice Game Jam.

What I Built

Bletchley's Longest Day is a browser-based cipher escape game set inside a fictional Bletchley Park night shift.

The player has to stop a U-boat convoy attack before dawn by clearing five rooms. Each room contains three escalating locks, so the full escape requires 15 solved puzzles. The game combines Caesar shifts, A1Z26 number decoding, Morse, anagrams, fragment ordering, a visible countdown timer, mistake penalties, hint penalties, account-based score saving, and a best-score leaderboard.

The solstice theme became the core dramatic clock: night is running out, first light is coming, and the player has to decode the final signal before dawn.

Video Demo

The demo shows the opening briefing, the three-lock room flow, the Gemini hint penalty, and the final victory state that only appears after all 15 locks are cleared.

Live game: https://bletchleys-longest-day.onrender.com

Code

Repository:

himanshu748 / bletchleys-longest-day

Bletchley's Longest Day

A browser escape-room puzzle game built around the June solstice: five Bletchley Park huts, fifteen locks, and twelve real-time minutes before dawn reaches the convoy.

Live game: https://bletchleys-longest-day.onrender.com

Gameplay

Clear five rooms before the mission clock reaches dawn.
Each room has three locks: cipher shifts, number codes, Morse bursts, anagrams, ordering, and final clearance.
Wrong answers cost time and score.
Hints are limited, penalized, and powered by Gemini when GEMINI_API_KEY is configured.
Final rank rewards speed, accuracy, streaks, and low hint usage.

AI And Assets

Gemini gemini-2.5-flash-lite powers contextual hint generation from the server-side /api/hint endpoint for signed-in players.
Guest players can play the full game and use standard built-in hints, but do not get Gemini analysis or leaderboard saving.
Higgsfield-generated images provide scene, evidence, operator dossier, dispatch, and defeat visuals.
The Gemini key is never exposed to browser JavaScript.

Auth And Leaderboard

Supabase Auth powers email/password…

View on GitHub

How I Built It

The game is a lightweight Node-served browser app. The front end is a hand-built HTML/CSS/JavaScript game surface, while server.js serves static files and protects the Gemini API key behind a server-side /api/hint endpoint.

The main design goal was to make the game feel like a tense intelligence desk rather than a generic puzzle page. Every room has atmosphere, evidence props, lock-specific copy, feedback states, and a timer that is always part of the pressure.

The puzzle structure was tuned around three ideas:

Three locks per room: each room has to be solved in stages, so the player earns the escape instead of clicking through one answer.
Time as score pressure: wrong answers and hints cost time, while clean solving preserves the best leaderboard run.
Guest mode vs signed-in mode: guests can play the full game, but Gemini-powered hints and saved leaderboard scores belong to authenticated players.

Google Gemini is used as a server-side hint officer. When a signed-in player asks for help, the game sends Gemini the active lock name, prompt, mechanism, visible puzzle text, failed attempts and hint level. Gemini then returns a short, question-specific nudge without revealing answer words. There is also an answer guard and fallback hint system so the game never depends blindly on model output.

Antigravity helped drive the build loop: implementing, checking, playtesting, tightening responsive UI and iterating on the final submission assets.

For hosting, the game runs on Render. Supabase handles authentication and leaderboard storage so each username has one best score rather than repeated leaderboard spam.

Prize Category

I am submitting for Best Google AI Usage because Gemini is not just decorative here. It is part of the gameplay economy: hints are question-specific, limited, penalized and unavailable to guests.
Antigravity helped drive the build loop: implementing, checking, playtesting, tightening responsive UI, and iterating on the final submission assets. Also veo is used through higgsfield for assets.

I am also submitting for Best Ode to Alan Turing. The game is built around codebreaking under time pressure, Bletchley Park atmosphere, wartime signals, and the feeling of solving small pieces of a larger intelligence picture before the world changes at dawn.

What I Am Proud Of

I like that the game does not treat AI as a free answer button. Gemini is useful, but it costs time and score. The best run still belongs to the player who thinks clearly under pressure.

The final shape feels like a compact escape room: readable enough to play on desktop or mobile, but hard enough that a clean 15/15 run feels earned.