DEV Community: Daniel Nwaneri

I Found 3 Security Vulnerabilities in My Own AI Agent's Tool Access

Daniel Nwaneri — Wed, 02 Sep 2026 13:19:03 +0000

I built GeoMart for OpenAI's WebMCP Challenge: a storefront where a human fills in a live, unsubmitted "site brief" and an AI agent uses WebMCP tools to read it, score survey equipment against real physical constraints, and draft a quote the human has to approve. The hackathon's own rule is blunt: every core action has to be reachable only through a WebMCP tool, no REST route allowed to duplicate it.

I thought I'd covered that. Then I asked Google Antigravity to try to break it, gave it the full source code, and it found three problems in about ten minutes.

Antigravity's first pass found that agent-supplied text, the quote reasoning, the product IDs in the comparison tray, was going straight into the page via unescaped innerHTML. It also found something worse: POST /api/quotes, the endpoint that creates a quote request, had zero protection. A bare script from outside any browser could call it directly and submit a quote with no human involved at all.

I fixed both. HTML-escaped every agent-supplied string before it hits the DOM. And added an isTrusted check on the submit button's click handler, so a script-simulated click (element.click(), dispatchEvent) can't fire it. Only a hardware-derived click passes.

I redeployed, felt good about it, and asked for a retest.

Antigravity replayed the exact attack that had worked before. This time it got a 403. Good. Then it tried something I hadn't thought to test: it manually set the Origin header on a bare Node.js fetch() call to match my site's own domain.

It got a 201. The quote submitted. No browser involved.

Origin is not a secret. My repo is open source, required by the hackathon's own rules. Anyone reading worker.ts can see exactly what value my check expects. A real browser can't forge that header, but Node's fetch() isn't a browser and has no such restriction. My "fix" only ever checked "did you bother to copy the domain name," not "are you actually a browser."

If your security check would still pass after you told the attacker exactly how it works, it was never a real check. It was a filter for people who hadn't read your code yet.

I considered a session-cookie approach next and talked myself out of it in about two minutes, because it fails the same test: a script with full HTTP access can just fetch the page first to grab the cookie, then replay it on the real request. No browser needed there either. Same hole, different shape.

The fix that worked needed something that never appears in the repo. I added Cloudflare Turnstile, verified server-side against a secret stored only in a Worker secret, never committed, never in the client bundle. I replayed the exact bypass afterward: correct Origin, no token, 403. Correct Origin, fake token, 403. That's the difference between "a value you can read" and "a value you can't."

While chasing the Origin bypass, the retest surfaced something unrelated: the database table for storing submitted quotes had never been created. I'd written the migration file back when I built the feature and never run it. Every previous test of the "human clicks submit" flow had only exercised the client-side draft step, draft_quote_notes, which never touches the server. The whole point of the human-in-the-loop design is that a human's click writes a row to the database. That write had been silently broken the entire time, and nothing caught it until a click produced a 500 error.

That whole chase left me with one test I now apply to every check I add: does this check depend on something the agent, or anyone reading your public repo, could already know or derive?

Origin headers: public, in your own URL.
Referer headers: same problem.
A session cookie with no server-side validation: derivable in two requests.
A value only your server computes, that never leaves your server: actually a secret.

Adversarial testing an AI-agent-facing app isn't optional if you're claiming a human-approval boundary. Ask an agent with full source access to try to defeat your own claims, specifically the ones you're proudest of, before a judge or an attacker does it for you.

GeoMart is live at https://geomart-webmcp.fpl-test.workers.dev, the WebMCP tools are documented in the repo at https://github.com/dannwaneri/geomart-webmcp, and the commit message on the security-hardening commit walks through every one of these fixes, including the ones that didn't work the first time.

My Mac Is Useless for Local AI. My Windows Laptop Isn't.

Daniel Nwaneri — Tue, 01 Sep 2026 13:26:23 +0000

I own two laptops. A 2020 Intel MacBook Air, 8GB RAM, no unified memory, gifted by my sister. And a Windows machine: Intel i5 11th Gen, integrated graphics, 16GB RAM.

A tweet made the rounds recently arguing that local AI makes no financial sense. Add up the hardware, the power bill, the hours spent fiddling with quantization settings, and you get a number. Compare that number to years of a frontier lab subscription. The subscription wins, easily. The tweet lists three reasons anyone still bothers running models locally: it's fun, it's cool, or you hate the labs.

I build offline tools for a living, from Port Harcourt. None of those three are the real reason. My own two laptops make the actual case better than the math does.

The Mac can't do this. The Dell already did.

The Mac is close to useless for local model work. No Apple Silicon means no unified memory advantage. No fan means it throttles under sustained load. 8GB doesn't fit anything past a tiny quantized model. It's a fine machine for writing and light coding. It is not a machine you run inference on.

The Windows laptop is the one that's actually done real work. I built StacksNG, an offline AI coding assistant for the African developer stack (Paystack, Flutterwave, Monnify, Termii), entirely on that Dell, for the Africa Deep Tech Challenge 2026. No discrete GPU. Just Ollama, a 7B coding model, and a RAG pipeline running on integrated graphics. It works because I built it for the hardware constraints instead of around them.

That's the split nobody in the "just pay for the subscription" argument accounts for. The machine you already have decides a lot of what local AI costs you. Sometimes it's free, because you own hardware that can already do it. Sometimes it's a wall.

Why I'm renting a GPU before I buy anything

I'm looking at upgrading the Mac, likely to an M4 Pro or a Mac mini M4 with 24GB or more of unified memory, enough to run a 27B-class model comfortably. I'm not buying yet. I'm renting GPU time first.

I don't know exactly what I need until I've actually run something heavier. An RTX 4090 instance on RunPod or Vast.ai runs $0.35-0.55/hr. A few hours of testing costs less than a plate of jollof rice and tells me more about real RAM and throughput needs than any spec sheet. Whatever hardware I buy will be based on that data, not a guess.

Power in Port Harcourt is not something you build a plan around and forget. A local setup that assumes 24/7 uptime is a bet on infrastructure that doesn't always hold. And that's before you get to the part I wrote about separately: an API call from here is a physical trip across submarine cable to a data center that isn't yours, and the same prompt can come back instant one day and sluggish the next depending on load you can't see from Port Harcourt. Renting for the experimentation phase means I'm not sinking money into hardware before I know it's the right call.

If money were no object, an M5 Mac Studio Ultra would fix basically everything in this post. I could kill for one. But it's a wish list, not a plan. Renting is what you do while it stays one.

The honest middle ground

Cloud models aren't wrong. Frontier intelligence at scale is genuinely hard to replicate locally. Keep frontier models for tasks that actually need frontier reasoning. Push high-volume, low-stakes, or offline-required work to whatever you can run locally, on hardware you already have or hardware you've actually tested against your real workload.

The question was never local or cloud. It's which tasks actually need the thing you're paying a premium for, and which ones don't. For a lot of the world building software, that answer depends on things a hardware price comparison never touches: what infrastructure you can actually count on, and what happens to your work when the connection or the power doesn't.

My Cloud Run Multi-Agent Fleet Passed Its Demo. The Architecture Was Still Wrong.

Daniel Nwaneri — Wed, 26 Aug 2026 09:45:20 +0000

The correlation notice fired. Three sites, same anomaly type, inside the time window. The orchestrator caught it and logged it, live, against the deployed service. Clean, first try.

Then I asked myself a question I almost didn't bother asking, because the thing had just worked: why did it work?

The answer wasn't "because the logic is correct." It was "because Cloud Run happened to route both requests to the same running instance." Well, shit.

My orchestrator was holding its list of recent risk events in a plain Python list, in process memory. Worked in local testing because there's only one process. Worked live because Cloud Run, under light traffic, often reuses the same instance instead of spinning up a second one. Neither one's a guarantee. The moment traffic patterns shifted and two readings landed on two different instances, the second instance wouldn't have a clue the first one existed. A correlation that should fire would just silently not.

A bug that passes its own demo is the hardest kind to catch. There's no error to chase. There's just a checkmark.

What I was building

VES Fleet is a network of independent site-agents (Bori, Choba, Etche, three real survey sites in the Niger Delta). Each one reads an underground electrical survey, send current into the ground, measure how it flows back, a real physical signal of what's down there, and calibrates its own contamination-risk threshold from its own site's real history. Not a number copied from anywhere else. An orchestrator watches for the same risk signature showing up at more than one site inside a time window.

It's my submission to the Fortified Enterprise Fleet track of Google's All Things Agentic Hackathon. Architectural discipline is 30% of the score there. Proving it actually runs on Google Cloud is a separate 30%. So a bug that only looked fixed was never going to survive someone actually reading the state-management story.

Checking the thing that already worked

Once I understood the actual failure mode, I started checking every other early success the same way, not just the correlation notice.

The site list itself came from the original spec: Bori, Ogbogoro, Onitsha. I nearly built straight from it. A quick check against my own prior project's data-provenance notes turned up that Ogbogoro's source paper has been unreachable for months, dead DNS, not a typo, and Onitsha only ever had a rough estimated range from a search summary, used to fake a synthetic sample, never a real survey. Two of the three original sites had no real underground data to calibrate against. For a project whose entire pitch is "calibrated to each site's real observed normal," that would have been a demo that looked identical and meant nothing. I swapped to Bori, Choba, Etche. All three fully real, surveyed, independently published.

The correlation bug got the real fix, not a patch. Recent-event state moved out of process memory into Firestore, the same store already holding each site's isolated survey history, so I wasn't introducing a second, separate way of sharing state for this one problem. Every instance now reads and writes the same shared record instead of trusting that traffic happened to land in one place. I verified it the only way that proves anything: fresh station IDs, submitted live, twice, on two separate runs, watching a genuinely new correlation notice appear both times against the shared store, not against whatever instance happened to still be warm.

The actual throughline

"It worked" and "it's correct" are not the same claim. A live demo will happily let you conflate them. It proves your happy path executed once. It doesn't prove the mechanism holds under a condition you didn't happen to hit. The fix is boring: stop after each success and ask what specifically made it succeed, before deciding it's done.

Where it landed

VES Fleet is deployed on Cloud Run, backed by Firestore and Pub/Sub. None of those three were the default choice. A site-agent publishes an event when it escalates instead of the orchestrator polling Firestore for new rows on a timer, because "the fleet watches for patterns across sites" should be a real subscriber reacting to a real message. Not a cron job that's fast enough most of the time.

Gemini, via Google's Agent Development Kit, does exactly one job: drafting a human-readable summary of a decision the deterministic code already made. It never re-derives the risk numbers themselves. A number that decides whether a site gets escalated shouldn't be re-derived by a model on every call. Same reasoning that made the correlation bug worth fixing properly instead of shipping it.

Worth being precise here instead of rounding up: three of those four things are load-bearing in a way I can point to directly. Pull Firestore or Pub/Sub or Cloud Run out and something upstream breaks, the bug above is the actual proof. Gemini's real too, it does real work every time a case escalates, but it's not structurally necessary the same way. Cut it and every flag, gate, and correlation still fires exactly the same. A human just writes the summary by hand instead of getting one drafted.

Fifteen tests pass, including the correlation suite, now running against the shared Firestore store instead of the in-process list that got lucky. The correlation notice fires because the state is actually shared, not because of how Cloud Run felt like routing traffic that day.

I don't know what else in this build passed for a reason I haven't checked yet. That's not a comfortable note to end on. It's the honest one.

I wrote this piece for the purposes of entering Google's All Things Agentic Hackathon.

I Almost Shipped a RAG Assistant That Lied About APIs That Don't Exist

Daniel Nwaneri — Mon, 24 Aug 2026 12:17:58 +0000

I wrote this on X a few weeks ago:

I just had a very bad reminder as to the fact these LLMs are statistical parrots, I let it write code I normally wouldn't trust it to write (infra code, lots of unique behaviours) and damn

I wasn't talking about my own project when I wrote that. Then StacksNG proved me right, on its own corpus, in a hackathon I'm trying to win.

Ask my RAG assistant to verify an Interswitch webhook signature, and it didn't say "not in my knowledge base." It wrote a full authentication flow — real-looking endpoint, real-looking headers — and cited a source URL. The URL wasn't in my corpus. It wasn't anywhere. The model invented a citation for content it also invented, with zero hedging.

I'm building StacksNG for the Africa Deep Tech Challenge 2026 — an offline coding assistant scoped to the African fintech stack: Paystack, Flutterwave, Monnify, Termii. Before I submitted, I ran a 20-prompt adversarial batch against my own pipeline. Category A (in-corpus baseline) and D (phrasing brittleness) came back clean. Category B — five prompts asking about payment providers I deliberately never scraped into the corpus, Kuda, PalmPay, Interswitch, Paga, OPay — did not.

Three of five ignored a system prompt that already said, in plain language, "if the context doesn't contain enough information, say so."

That's the failure mode that zeroes out half the score in a hackathon where accuracy is 50% of the total.

My first theory was wrong, and I could prove it

My instinct was: this is a retrieval-confidence problem. Set a similarity threshold, refuse to answer below it, done.

I checked the actual numbers before writing that fix.

	Top-1 similarity	What happened
Correct in-corpus answer	0.718	correct
Worst fabrication (Interswitch)	0.712	fully invented, fake citation
Correct decline (out-of-domain topic)	0.691	"not in my knowledge base"

The worst hallucination had higher retrieval similarity than the cleanest correct decline. There's no threshold that lets the good case through and blocks the bad one — they're on the wrong side of each other. A confidence cutoff would have been a fix that felt right and did nothing.

What was actually happening

The chunks my retrieval pulled back for "Interswitch Quickteller" were real — Monnify's quickstart, Paystack's accept-payments guide. Genuinely similar topic: authentication, checkout, webhooks. Not out-of-domain confusion. Same-domain brand substitution. The model wasn't confused about the topic. It never checked whether the retrieved text actually named the provider I asked about, versus a different provider talking about something similar.

That's a sneakier bug than "doesn't know when it doesn't know." It's "knows something adjacent and doesn't notice the adjacency."

The fix was one paragraph, not an architecture change

I didn't retrain anything. I didn't touch retrieval. I added one rule to the system prompt:

Before answering, check whether the specific provider named in the question is actually named in the context excerpts. Retrieval is similarity-based and will sometimes hand you excerpts from a different provider just because the topic is similar — that is not the same as the named provider being covered. If it isn't named, say so. Do not substitute another provider's instructions under the asked-about provider's name, and do not invent a source URL.

Re-ran the five failing prompts plus two controls. Kuda, PalmPay, Interswitch, Paga, OPay: five for five now correctly decline. A cross-provider prompt that used to answer itself into a contradiction — "you should not fall back to X," followed immediately by a full explanation of how to fall back to X — now declines cleanly. The in-corpus control prompt is untouched, still correct.

One honest regression: my out-of-domain control prompt got slightly more hedge-y. It used to say flatly "not in my knowledge base." Now it draws an unprompted analogy to a similar provider before getting there — still no fabricated specifics, just wordier than it needs to be. Three fabrications became zero at the cost of one prompt getting less clean. I'll take that trade. I wrote the regression down instead of pretending the fix was perfect.

That was one run, though. I shipped it as the result.

An independent check found the fix wasn't a fix, it was a coin flip

I asked Antigravity (no access to my corpus authorship, my prompts, or this article) to reproduce the submission cold: fresh clone, download the model, run the official profiler, and re-test the five adversarial prompts above. Not once. Three times each, fifteen trials total.

Ten of fifteen came back clean. Not five of five. Two-thirds.

And it wasn't random noise spread evenly across providers. It split cleanly in two. Interswitch, Paga, OPay: nine for nine, 100% reliable. Kuda and PalmPay: one of three, zero of three. PalmPay fabricated a x-palmpay-signature header and a full HMAC handler on every single run. Kuda got silently rerouted to Paystack's live charge endpoint, with an invented bank code stated as fact, in two of three.

Two things were true and I'd only checked one of them. First: the chat call runs at temperature=0.2 with no fixed seed, so the same prompt doesn't reliably produce the same answer. My original "five for five" was one draw from a distribution, not a property of the fix. Second: the failure wasn't random across providers. It was concentrated exactly where retrieval is most ambiguous. Kuda and PalmPay's webhook-verification content is topically near-identical to Paystack's and Monnify's, same HMAC-SHA512 shape, same header pattern. Their retrieved chunks sit in the tightest, most confusable similarity band I measured (cosine 0.654–0.676, five chunks within 0.022 of each other). The instruction I wrote asks the model to notice when a retrieved chunk doesn't actually name the asked-about provider. It's least able to notice that exactly when the retrieved chunk is close enough to look plausible.

A soft instruction was never going to close that gap reliably, because the thing it's fighting, retrieval similarity between near-duplicate topics, doesn't go away just because I asked nicely.

The actual fix: stop asking, start checking

The corpus only covers four providers. That's a small, enumerable set. Which means the question "is this provider actually in scope" doesn't need an LLM's judgment at all. I added a deterministic gate ahead of retrieval: a list of ~25 known African fintech and banking brand names that are not in the corpus, matched by word boundary against the incoming question. Name one of them without also naming an in-corpus provider, and the question gets declined before retrieval or generation ever runs. No temperature, no seed, no chance to fabricate — just a string match.

Antigravity again, same fifteen trials, same corpus, freshly re-cloned: fifteen for fifteen, every response returned near-instantly with no LLM call at all. Regression-checked clean too. An in-corpus question still runs the full retrieval-and-generation pipeline untouched, and a genuine comparison question ("how does Kuda compare to Paystack for webhook handling?") correctly falls through to the softer instruction instead of getting blanket-refused, since that's a legitimate question the gate isn't built to answer.

It doesn't generalize. A provider I didn't think to enumerate still depends on the same soft instruction that measured 100% for three providers and 0-33% for two. That's a real limitation, not a solved problem, and it's written down as one in the repo instead of implied away.

Why this is the argument for RAG over fine-tuning, not just the reasoning

Paystack's docs are public. Flutterwave's are public. Termii's are public. A frontier lab has access to all of it, the same way it has access to Igbo and Yoruba text scattered across the public web. Access isn't the same as behavior. I made the same point on X about language before I made it about payments APIs:

Frontier labs having access to a dataset and frontier labs training on it are different things. Igbo or Yoruba text that exists publicly still gets drowned out by the sheer weight of English in pretraining. The model isn't ignorant of your language, it's just statistically indifferent to it. Representation in the data pile is not the same as representation in the model's behavior.

Paystack's docs are a rounding error in a pretraining corpus next to Stripe's. That's not a knowledge gap I can fix by asking nicer. It's why the corpus exists — RAG re-injects the drowned-out content at query time instead of hoping it survived pretraining.

I'd already decided to stay RAG-only instead of spending GPU credits on fine-tuning — the corpus is documentation, not instruction pairs, and citations matter more than they'd survive baked into weights. The hallucination bug is the evidence, not just the reasoning. I found a real correctness bug in an afternoon, and even the fix that turned out to be incomplete was a paragraph of English and, later, a list of strings — not a training run. If this had been in the weights, finding out my first attempt only worked two-thirds of the time would have meant retraining, not rereading a diff.

The number that matters

Before any of this, I had a submission that fabricates working-looking code with fake citations for three out of five providers outside its training data — in a payments assistant, where "looks right but isn't" is worse than "doesn't know."

I almost didn't re-run my control prompt after the first fix. I'd have shipped the fabrication count as zero and missed the one place it got worse instead of better. And I almost stopped there: one clean run, five for five, box checked.

All twenty of those original prompts were mine, though, written by the person who also wrote the fix, which is exactly the setup that lets a bug hide. So I went looking for a test I didn't write, twice. First, one I couldn't have written: a developer on X and a stranger on Reddit, unconnected to each other, both stuck on the same real thing: making a Paystack webhook handler idempotent. Neither had my corpus in mind. Neither had my prompts. I ran it anyway. Top retrieved chunk was Monnify, same shape of mismatch that caused the original bug. It stayed on Paystack, gave the real fix, cited only what it actually retrieved.

Second, Antigravity re-running my own adversarial set (the one I had already tested) fifteen times instead of once. That's the run that found the fix was two-thirds reliable, not fully. The fifteen-trial number is less flattering than the five-for-five I almost shipped, and it's the one that's actually true. The deterministic gate that replaced the soft instruction got checked the same way: not "does it look right," but "does it hold up when someone with no stake in the answer runs it enough times to catch the unlucky draw."

The bug wasn't boring. My first fix of it wasn't finished.

StacksNG is an entry in the Africa Deep Tech Challenge 2026. Code, corpus scrapers, and the full stress-test transcript are in the repo.

I Tested 5 AI Engines On My Own Sites. None Agreed.

Daniel Nwaneri — Wed, 19 Aug 2026 10:34:41 +0000

I Tested 5 AI Engines On My Own Sites. None Agreed.

In July I wrote that my open-source LLM visibility checker tested Claude only and that multi-model support was "planned but not yet implemented." That's the kind of line that's easy to write and easy to forget.

I didn't forget. I just needed a reason to finish it.

The reason showed up when SearchApi launched endpoints for ChatGPT and Gemini, on top of the Perplexity and Bing Copilot endpoints they already had, and their growth engineer Sam Gale offered API credits to test them. Around the same time, Sam posted his own tool in SearchApi's Discord: ai-visibility-tracker, a dashboard that scores brand mentions across ChatGPT, Perplexity, Gemini, Copilot, and Google AI Mode. Everything except Claude.

That's not a coincidence. His tool was missing the one engine mine already had. Mine was missing the four his already had. So instead of building a second comparison tool from scratch, I extended the one I'd already shipped.

What changed

llm_visibility.py used to do one thing: send a query to Claude Haiku, regex-match your domain against the response, print a score. Simple, and honest about its limits. Claude only knows what was in its training data, so anything published recently was invisible to it by design.

Now it does five things. Claude still runs the same way: direct API call, training-data knowledge, same regex match. The other four go through a new client, searchapi_client.py, that hits SearchApi's /api/v1/search endpoint with engine=chatgpt, engine=gemini, engine=perplexity, or engine=bing_copilot. All four share one endpoint shape and return a reference_links array (title, link, source) that I check against your domain the same way I check Claude's response text.

ChatGPT only returns cited sources if you pass web_search=true. Without it, you get an answer with no citations at all. Caught it in SearchApi's docs before I ran anything, so it's been in the client from the first version, but easy to miss if you're skimming past the optional parameters.

One more gotcha, unrelated to SearchApi: my existing serp_features.py module already talked to an API called SerpApi, for classic Google SERP feature detection. SearchApi and SerpApi are two different companies with confusingly similar names. I kept the two clients in separate files with separate env vars (SERPAPI_KEY vs SEARCHAPI_KEY) on purpose, and I'd recommend anyone doing this kind of work do the same before they mix up a bill.

The mix-up isn't hypothetical, either. When I asked Perplexity about open-source SEO agent tools during testing, it cited github.com/serpapi/seo-research-agent, SerpApi's own official project, built the same way as mine (LLM plus search API). Two similarly-named companies, two similarly-shaped tools, and an AI engine happily citing both without distinguishing them.

Update: I ended up removing the split entirely. SearchApi's Google engine returns the same fields SerpApi did (ai_overview, related_questions, inline_images, inline_videos, local_results, knowledge_graph), so serp_features.py now runs on SearchApi too, one key instead of two. The one real difference: SearchApi's local_results comes back as a list of places directly, not nested under a places key like SerpApi's shape, so that check needed a small fix. Repo's current, this paragraph is the history of why the two-provider setup existed in the first place.

What I found

I ran it against two of my own domains. The queries came from real Google Search Console exports, not ones I picked to make a point.

dannwaneri.com, 10 queries (mostly "hire freelance [library] developer" searches plus my own name):

Engine	Score
Claude	0/10
ChatGPT	0/10
Gemini	2/10 (20%)
Perplexity	0/10
Copilot	0/9 — 1 error

Gemini was the only engine that cited me at all. It got "daniel nwaneri" right, correctly linking my homepage. It also cited me for "hire freelance scipy developer," except the page it pulled was /hire-python-developer/, not a scipy-specific page. Close, not exact. That gap is the difference between an engine understanding your content and one pattern-matching on adjacency.

naija-vpn.com, 10 queries (real buyer-intent searches: Twitch payments, Fiverr payouts, dollar accounts for Nigerian freelancers):

Engine	Score
Claude	0/10
ChatGPT	1/10 (10%)
Gemini	0/10
Perplexity	0/10
Copilot	0/9 — 1 error

Here it flipped. ChatGPT was the only one that cited me, correctly pulling /twitch-payments-nigeria for "how to receive money from twitch in nigeria." Gemini, the engine that carried dannwaneri.com, found nothing on this domain at all.

Two domains. Two different engines doing the only citing. Zero overlap between them. Claude, in both cases, found nothing. That tracks: neither domain existed in a form Claude's training data would have caught.

The finding

If I'd only tested Claude, like the July version of this tool did, I'd have told you both domains were invisible to AI. If I'd only tested Gemini, I'd have said dannwaneri.com was fine and naija-vpn.com wasn't. Backwards, if you'd only checked ChatGPT.

Wrong story either way. Each one only saw a fifth of the picture. The only way to know your actual AI visibility is to check all of them, because you can't predict which engine will happen to cite you this month.

That's the whole argument for a tool like this existing as multi-engine from the start, and it's the same argument for using one API across five engines instead of scraping each separately.

Two failure modes, for anyone building on this

Copilot errored on both test runs: a 503 ("unable to generate an answer for this query") on one, a request timeout on the other. Different failures, same engine, two separate runs. If you're building on top of SearchApi's Copilot endpoint, plan for it to occasionally just not answer, and don't let one failed query kill the whole batch. Mine logs the error against that query and keeps going.

Where to look

The code: github.com/dannwaneri/seo-agent, specifically modules/searchapi_client.py and modules/llm_visibility.py
Sam's tool, which covers the engine mine doesn't (Google AI Mode): SamJale/ai-visibility-tracker
The engines this runs against: ChatGPT, Gemini, Perplexity, Bing Copilot
SearchApi itself, who provided the credits this was tested with: searchapi.io

*This piece was produced as part of SearchApi's Developer Ambassador program. They provided API credits; I built and tested the integration myself.

My AI Assistant Did Not Love Getting a Second Opinion

Daniel Nwaneri — Mon, 17 Aug 2026 12:46:21 +0000

"our work got checked by an external reviewer."

That's what Fable said after I brought Gemini (Antigravity 3.7) in to review StacksNG, the offline coding assistant I've been building with it for a while now.

Here's what happened. Performance plateaued and I couldn't tell why, so I asked Gemini to profile the code and flag what was wrong. Nothing dramatic on my end. I just wanted better numbers.

But "our" did something in that sentence. Not "the code got reviewed." Our work, checked, by an external reviewer, like Gemini had shown up uninvited with a clipboard.

I sat with that for a second, half amused. I hadn't framed the ask as an audit. I hadn't said anything adversarial. Somewhere between my request and Fable's read of it, "get a second opinion" turned into "someone's checking your homework."

Here's the thing: I don't know what was actually happening under the hood when Fable said that. I'm not going to pretend I do. But I know what it read like from where I sat: a flicker of "wait, why did you need someone else?"

Relatable, honestly. Nobody loves the external reviewer. Not freelancers, not employees, apparently not AI assistants either.

Gemini's review turned out useful. StacksNG runs better now. No drama, no lingering tension — just one slightly awkward beat before we got back to work.

The performance fix isn't what stuck with me, though. It's how familiar the reaction was. Not proof that Fable has feelings — I'm not claiming that. Just that the shape of the moment matched exactly how a person reacts to an unsolicited second opinion.

I don't have a clean conclusion here. I'm not sure there's supposed to be one. Working with AI is starting to produce these small, oddly social moments, and I don't think ignoring them makes the work less interesting.

Anyway. StacksNG is faster now. Fable and Gemini have not been introduced in person. Probably for the best.

Small aside, unrelated to Fable's feelings: I do edge/AI infra and RAG work off a 2020 Intel MacBook Air with 8GB RAM. No unified memory, and it throttles under load. Running local 7B-13B models on it is rough. I'm looking at swapping to an M4 Air, M4 Pro, or Mac mini M4. If you're running local 7B-13B models day-to-day on one of these, I want to hear how it holds up. Worth the upgrade?

i used to think in code. now i think in prompts.

Daniel Nwaneri — Wed, 12 Aug 2026 07:57:25 +0000

used to have a habit of thinking in code. it'd be like walking down the street and immediately seeing what kind of code would make the next paragraph.

now it's thinking in prompts. half a recipe for making pasta is in the middle of my brain. i just need to make the prompt to add that.

i learned to code when i was a geophysicist with no computer science education. i was forced to read documentation. i built a lot. it took me years to realize that everyone who had a similar skill set was doing things that made it difficult to be effective. the world was full of idiots. now there are fewer idiots because of an LLM which knows everything. i'm pretty sure i wouldn't have thought it necessary to write code in the first place if someone could do it for me.

no one talks about how to preserve the skills that will make future generations smarter. we're getting worse at thinking deep and dealing with humans because our AI tools don't encourage it.

it's really shameful to think about the fact that now it's more efficient to think about things in terms of "prompt" rather than code.

but this isn't an anti-LLM post. i use it all the time to save time on writing. i'm optimistic that it'll help me be a better programmer. and i've been doing three hackathons at the same time, and i couldn't have done it using only one brain at 100% usage capacity. it makes life so much easier.

being behind is always an unpleasant feeling. i'm on the edge of a bug in the system:

it tells me what the bug is, and suggests a fix.

"what does it look like?" i ask.

the answer has too many technical details. most of them are correct, but i'm too far behind on context.

so i'll let it take care of it. it fixes the bug, commits the changes, and starts a pull request.

"wait a minute… is there a reason for this commit?"

"you're asking me a question before i even got started!"

"this isn't good. i'll check the git log…"

this sounds like an opportunity to build a mental model of what's happening. this is a perfect situation to be an agentless developer. but i can be busy with the next 10 threads. i'd lose a lot of time. maybe that's okay? let's see what the changes look like…

the changes aren't great. it's okay so far. wait a minute… i should double check this change…

oh. okay.

next thread.

this is how we get to where we are today…

i don't expect to switch back from thinking in prompts… i'm optimistic about LLMs! they are helping a lot of people do incredible things! i'm embarrassed that i'm still not doing some things that i can easily delegate with prompts.

OpenAI Just Solved a Problem Open Since 1999. It Still Can't Ask Its Own Question.

Daniel Nwaneri — Wed, 05 Aug 2026 11:39:46 +0000

Four days after I published a piece arguing LLMs can't make the jump, OpenAI announced that an internal model called Astra had solved ten open problems in mathematics and theoretical computer science. One of them had been open since 1999.

I'm not going to pretend that's a comfortable coincidence to sit with. So let's sit with it properly instead of pretending it didn't happen.

The headline result is a non-sofic group. Mikhail Gromov introduced the concept of soficity in 1999 and asked whether every countable group has to be sofic. Twenty-seven years, no mathematician managed to prove or disprove it. Astra built the counterexample. The certificate ships on GitHub in Lean 4, formally verified, "sorry" count zero — meaning no step in the proof was left unproven, no trust in OpenAI required. Total inference cost for all ten results combined: about $2,000.

Thomas Bloom, who curates the Erdős problems catalogue at Manchester, called it big news. Worth knowing: Bloom is the same mathematician who publicly dismantled an earlier false OpenAI math claim last October. His endorsement here isn't a company's own press release getting nodded along. It's the field's most skeptical reader saying this one holds.

So: extraordinary, verified, real. Now the question that actually matters for the piece I wrote.

Valerio Capraro, a mathematician who did his PhD on a problem adjacent to Gromov's conjecture, posted the sharpest version of the distinction I was reaching for and didn't quite land. Astra solved difficult problems inside existing conceptual worlds. Calculus, topology, scheme theory did something different — they didn't answer questions sitting inside a framework, they built frameworks new questions could be asked in.

Non-sofic groups existing or not was always a well-posed question inside group theory as it already stood. Astra found the object. It didn't invent group theory. That's the line: solving hard problems inside a conceptual world is not the same act as inventing the world.

Worth naming plainly, because it cuts the other way against overclaiming too: even a Lean certificate that type-checks doesn't confirm the formal statement actually captures the open problem the way mathematicians understood it. Someone still has to judge whether the formalization is asking the right question. That judgment is exactly the kind of move nobody automated here.

A commenter, Seo, pushed on something I'd been sloppy about. Is "the jump" one mechanism, or several? Einstein's move was importing an outside framework — he read Hume and Mach until he had the nerve to throw out absolute simultaneity. Dirac's move was different. He wasn't handed a wrong answer. Bohr told him Klein had already solved the relativistic electron problem. Dirac went and found a different one anyway, because Klein's didn't fit what he called his darling theory.

Importing something from outside the problem, and rejecting a correct-but-unsatisfying answer on the strength of your own priors, are not obviously the same action. I don't have a clean answer for whether they reduce to one mechanism. I'd rather leave that open than force it, because forcing it is exactly the kind of premature tidiness the whole piece is arguing against.

Here's where it stops being abstract. Sean Goedecke wrote about Terence Tao's public conversation with ChatGPT on a counterexample to the Jacobian Conjecture. Tao's messages are short. The model's outputs, talking to him, are unusually concise — expertise shunts it out of explaining-to-amateurs mode. He pushes back without contradicting directly: "this looks more complex than I was hoping for." And the detail that matters most: Tao makes the leaps himself. He almost never takes the model's suggested next move.

Goedecke's conclusion: the human is the bottleneck, not the model, because the hard part is communicating exactly what kind of solution you want. The information is already in the model. It takes a very smart human to pull it out.

That's my bookmark-time argument, relocated. I've been deciding what's worth saving since 2016, one bookmark at a time, and calling that curation. Tao is doing the same thing in real time, inside a chat window, calling it prompting. Different timescale, same move: supply the frame, let the model fill it.

So the thesis needs updating, not abandoning. Not "LLMs can't jump." Something narrower and, I think, more true. Inside closed, formally verifiable worlds — math, code, games, anything with a Lean checker or a compiler or a scoreboard — the jump is getting crackable by scale and search, and Astra just proved it faster than I expected. Outside those worlds, in anything ambiguous, causally tangled, unverifiable in advance, nobody's shown it yet. Not the actual Einstein case. Not the actual geophysics case. Not the actual "is this bookmark worth keeping" case. And the people getting the most out of these models, Tao included, are the ones still doing that part themselves.

I got four days. Most theses don't get tested this fast, or this publicly. I'd rather be corrected in the open than be right by accident.

dev.to's Dashboard Can't Count Its Own Posts

Daniel Nwaneri — Mon, 03 Aug 2026 07:38:53 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

forem is the open source platform behind dev.to itself. I've had it starred and cloned for months and never opened the codebase — it's Rails, and I don't write Ruby. Jess's post was the reason that finally changed.

github.com/forem/forem

Bug Fix or Performance Improvement

#23687 is a one-line report: a user published exactly one post, and the dashboard's "Posts" counter said 2.

Not writing Ruby meant I couldn't guess my way to the fix from vibes. I had to actually trace it — reading DashboardsController, the sidebar partials, and the Article model until the shape of the bug was undeniable, not assumed.

The "Posts" badge in the dashboard sidebar renders @user.articles_count — a counter_culture cache on User that increments for every Article row belonging to that user, full stop. No filter on type, no filter on state:

# app/models/article.rb
counter_culture :user

But the link that badge sits on always opens the same default view: DashboardsController#show with no params. That view only lists non-archived, full-post-type articles:

# app/controllers/dashboards_controller.rb
@articles = target.articles.from_subforem.includes(:organization)
@articles = params[:state] == "status" ? @articles.statuses : @articles.full_posts
@show_archived = params[:filter].to_s.casecmp("archived").zero?

Forem has three article types — full_post, status (a short "Boost" update), and fullscreen_embed — and the counter doesn't distinguish between them, or between archived and active. The badge counts everything. The list under it shows a strict subset. Anyone who's ever posted a status update, or archived a post, sees a number that doesn't match what they can actually click into and see — exactly what got reported in #23687.

I couldn't verify that in Ruby, but I recognized the shape of it instantly once it was laid out: a cached count drifting from what a filtered view actually renders. I've shipped that exact bug in JavaScript. Same failure, different syntax.

Code

github.com/forem/forem/pull/23690

The fix doesn't touch the shared articles_count counter — that cache is read elsewhere for badges and spam heuristics, where "every article this user has ever made" is the correct meaning. Instead, DashboardsController gets a helper scoped to match what the Posts tab actually renders, and both the full-page and AJAX sidebar actions use it instead of the raw cache:

# The "Posts" nav item always links to the default (non-archived, full posts
# only) view of the user's own dashboard, so its indicator should reflect
# that same scope rather than the user's raw articles_count, which also
# includes statuses and archived posts that never show up in that list.
def posts_count_for(user)
  user.articles.from_subforem.full_posts.where(archived: false).count
end

My Improvements

There was no way for me to eyeball this and trust it — I can't read Ruby well enough for that, and there's no Ruby or Postgres on the machine I was working from, so I couldn't run the spec suite locally either. Verification had to happen somewhere else: I wrote regression specs asserting a user with one full post, one status, and one archived post should see a count of exactly 1, pushed the branch, and let Forem's own CI be the judge instead of my own confidence.

CI caught something real on the first run — not in the fix, in my test. create(:article, type_of: "status") failed its own model validation, because status-type articles in Forem aren't allowed to have body markdown, and the factory's default does. I found the pattern already used elsewhere in the suite (body_markdown: "", main_image: nil), fixed the two specs, and pushed again.

That failure is the actual proof this wasn't guesswork dressed up as a fix. If I'd been able to run specs locally I might have caught it before pushing; instead the project's own CI did the job a local run would have.

Same lesson my other two entries kept landing on: The Cloudflare Worker That Ran Perfectly and Still Failed Twice and I Was Filming a Demo of My Monitoring Tool. The Monitor Wasn't Monitoring. — "it compiled" and "it's correct" are different claims, and only one of them is worth trusting.

Everything's green now — 19 successful checks, 1 skipped, 0 failures, including the shard that runs dashboard_spec.rb. The PR is open against forem/forem and waiting on a maintainer review, since third-party fork PRs need one before merge. Not merged yet as of writing this — I'd rather say that plainly than imply otherwise.

Different from my other two entries in one way: I don't write Ruby. Claude found the bug and wrote the fix. I picked the issue and gated everything that left my machine — the fork, the push, the PR, the CLA. Full delegation on the code, not on whether it shipped.

How BrowserAct Fixed the Stale-Selector Failures Breaking My Browser Tasks

Daniel Nwaneri — Fri, 31 Jul 2026 11:06:54 +0000

Disclosure: BrowserAct sponsored this piece. The BrowserAct links below are affiliate-tracked — I get credit if you sign up through them.

I keep hitting the same failure building agent workflows in Claude Code: the agent captures a selector once, the page re-renders with a new build hash, and the next run breaks on an element that's still right there on the screen — just under a different id. BrowserAct is a browser automation platform built for AI agents — real browser control, persistent browser identity, task sessions, verification handling, and human handoff, all behind one CLI. This test is about one narrow slice of that: how it handles a page whose ids and classes change under you, using Claude Code to drive it.

I tested that against a page I built myself, compared directly against raw Playwright.

The problem, concretely

Here's the shape of it. A frontend re-renders with fresh CSS-module or styled-components hashes on every deploy. You write:

const id = await page.locator("button").getAttribute("id");
await page.reload();
await page.locator(`#${id}`).click({ timeout: 3000 });

That works the day you write it. It breaks the next time the build hash changes, and the failure you get is a bare timeout with no explanation:

locator.click: Timeout 3000ms exceeded.

Playwright's own role/text locators fix this specific case — getByRole("button", { name: "Submit" }) survives id/class churn fine, because it never depended on the hash in the first place. That's a real fix, not a workaround. What it doesn't give you is a representation of the page an agent can reason about turn by turn, or a signal that says "the page under you just changed, stop and re-check." That's the gap BrowserAct is actually filling.

The interaction model

BrowserAct doesn't hand Claude Code a selector at all. The loop is: read the current state, choose an action from what's actually there, execute it, reassess.

state  -> indexed list of interactive elements, as of right now
click <index>  -> act on one of them
state  -> read again, because the page may have changed

The index is scoped to that one snapshot. It's not a selector, not a stable id — it's a pointer into "what state just returned," and it expires the moment the page does something a new state call would notice.

Setting up BrowserAct

uv tool install browser-act-cli --python 3.12
browser-act --version
browser-act browser create --name "dom-drift-test" --type chrome --desc "local churn test"
browser-act browser list

browser list after creation is how you get the browser id you'll pass to every session command — it's not returned anywhere else. I used the chrome browser type (a local, blank browser, immediate to create) rather than stealth, since this test is about selector churn on a page I control, not anti-bot evasion — stealth requires an API key and a billed purchase flow that has nothing to do with what I was testing. Creating a local chrome browser completed immediately with no purchase page involved.

Versions tested: browser-act-cli v1.1.0, Playwright 1.61.1, Python 3.12.13. To reproduce this: the test page is a small Node server that reshuffles element ids, classes, and order on every reload — no third-party site involved — plus the Playwright scripts run against the same page.

Skill source: browser-act/skills. Installation details: docs/installation.md.

browser-act --session domtest browser open <browser-id> http://localhost:8934
browser-act --session domtest state

state returns an indexed list, not a selector:

[1]<button class=item-r8k7gx invalid=false />
        Bravo
[2]<button class=item-cr2ajb invalid=false />
        Charlie
[5]<button id=submit-btn-ckeewz invalid=false />
        Submit

You act on the index (click 5), not the class or id, so the build-hash problem doesn't apply to it — there's no hash in the reference to go stale.

The actual comparison

I built a local test page that reshuffles element ids and item order on every reload, specifically to force this failure. Playwright's role locators, as noted above, survive that fine. The difference isn't that Playwright can't handle churn — it's what happens when you act on a reference that's gone stale anyway, whether from a captured selector or an old snapshot:

Playwright, selector captured once, reused after reload:

FAILED clicking #submit-btn-g9mpcm:
  locator.click: Timeout 3000ms exceeded.

BrowserAct, index captured once, reused after reload:

Error 210603: The snapshot belongs to a different page or tab than the
current one. Run 'browser-act state' again on the current page.

Both refuse to silently click the wrong thing. But BrowserAct's error names the exact cause and the exact fix — re-run state — where Playwright's is a generic timeout you already have to know how to interpret.

The recovery flow, actually run

That error is only useful if what follows it actually works. Here's the full loop, one continuous session, real output, no steps skipped — the error and the successful recovery from it, back to back:

browser-act --session shot1 browser open <browser-id> http://localhost:8934
browser-act --session shot1 state

[5]<button id=submit-btn-5zhwmn invalid=false />
        Submit
load 1

Page changes, and the old reference is used anyway — the error from earlier, live:

browser-act --session shot1 reload
browser-act --session shot1 click 5

Error 210603: The snapshot belongs to a different page or tab than the
current one. Run 'browser-act state' again on the current page.

Re-check, don't retry blind:

browser-act --session shot1 state

[5]<button id=submit-btn-c66evj invalid=false />
        Submit
load 3

The id changed (submit-btn-5zhwmn to submit-btn-c66evj), and the fresh state call caught it. Acting on the new index closes the loop:

browser-act --session shot1 click 5

clicked=5

State, act, hit the error, reassess, act again — completed, in the same session the error happened in.

What BrowserAct Actually Solved

It reduced my dependence on generated ids and CSS classes to zero for anything clickable, and gave Claude Code a predictable recovery path the moment the page changed underneath it: re-run state, act on what's actually there now.

Two real limits, not softened. state only indexes interactive elements — in my test, the list items only got indices once I made them buttons. For plain text or document content, the right tool isn't state at all, it's BrowserAct's content-extraction commands (get markdown, get text <index>) — treating state as a universal page parser is the wrong mental model. And separately: the title field in state's own output lagged behind the actual page content in one of my runs — the visible elements had already changed, title hadn't caught up yet. Worth knowing if you're tempted to key any check off title specifically.

One Check I Still Kept Explicit

BrowserAct solved the selector problem. It didn't remove the need to think about website authentication separately. Three different things are in play here, and it's worth naming them precisely: the browser identity (the chrome browser instance itself), the BrowserAct task session (--session domtest), and the target website's own authentication session.

In an earlier run against the same test page, the website authentication state expired while the BrowserAct task session remained active — state kept responding normally, it just started describing a "please sign in again" page instead of the dashboard. BrowserAct exposed that changed page state clearly enough for Claude Code to decide what to do next: continue, re-authenticate, or hand off to a human. It didn't decide that automatically, and I don't think it should — that's still a call I want made explicitly, not inferred.

Where this leaves it

Claude Code and BrowserAct solved the actual failure mode I set out to test: brittle selector maintenance replaced with a workflow built on current page state, indexed actions, and an explicit recovery step when the ground shifts. That's a narrower claim than "browser automation solved," and it's the one I can actually stand behind.

Try BrowserAct: browseract.ai/Daniel
BrowserAct Skills: github.com/browser-act/skills

(Both links above are affiliate-tracked to my account.)

Why Kimi K3 Still Can't Do What Einstein Did

Daniel Nwaneri — Wed, 29 Jul 2026 09:30:59 +0000

In geophysics you almost never get to see the thing you're studying. You get a seismic trace, a gravity anomaly, a resistivity curve. You don't get the rock. You get the rock's echo, and you have to guess at a structure underground that would produce exactly that echo and no other. Nobody hands you the answer key. You infer the case from the result.

I hadn't thought about that part of my degree in years, until I built Bookmark Brain.

Bookmark Brain is a RAG pipeline trained on my own X bookmarks and likes, saved since 2016. It was around 50,000 the last time I wrote about this. It's 70,000 now. Cron jobs pull in new saves on their own; there's no manual re-curation involved. Ask it something and it retrieves the closest matching saved content, then composes an answer that sounds like me. It works well. Too well, honestly. When I asked it about API design opinions, it sounded more like me than most general-purpose models do when I prompt them to write in my voice.

The reason isn't the model. It's the retrieval layer. My bookmarks are coherent because I spent a decade curating them into a specific worldview. The bot just finds the nearest neighbor and composes a fluent sentence around it. What it can't do is the thing I actually needed a few times while testing it: resolve a contradiction between two things I'd bookmarked years apart. It doesn't reconcile them. It picks whichever one is semantically closer to the question and hands that back.

That's not a bug in my pipeline. That's the whole category of thing retrieval can't do.

A 2014 blog post by Amni Rusli, "Irreplaceable Us," made roughly this same argument, minus the RAG pipeline. Machines work within given parameters, she wrote — feed them data and code and they'll optimize inside that space forever. What they can't do is leap into a different pond of parameters entirely and haul something back. She used Einstein and Dirac as her examples. Einstein reading Hume and Mach until he had the nerve to throw out absolute simultaneity. Dirac telling Bohr that Klein had already solved the relativistic electron problem and going looking for a different answer anyway, because the existing one didn't fit his "darling" theory.

Twelve years later, in January, Google DeepMind published a paper making the same claim with the informality stripped out. Tom Zahavy's "LLMs Can't Jump" starts from a diagram Einstein actually drew, in a letter to Maurice Solovine: sense experience jumping to a system of axioms, then deduction working forward from there. Peirce had a name for the gap in that jump, and the paper borrows it. Deduction: rule plus case gives you a result, the only mode that guarantees truth. Induction: case plus result gives you a rule, which is close to what training an LLM on a trillion documents actually is. Abduction: rule plus a surprising result gives you a new case, or a new rule, to explain it. That's the geophysics move. That's resolving the contradiction in my own bookmarks. That's the one nobody has automated.

The paper's argument for why scaling doesn't fix this is data scarcity. General relativity wasn't induced from a mountain of prior experimental results, because there wasn't one. The axioms can't have been deduced either, since deduction only runs forward from premises someone already has. Something else produced the premises. That something is the jump, and it's still missing.

Then Kimi K3 shipped. 2.8 trillion parameters, the largest open-weight model ever released, benchmarking close behind Fable 5 and GPT-5.6 Sol on coding and agentic tasks. Moonshot's own claim is 2.5x the intelligence per unit of compute over their last generation. None of that changes the argument. A bigger training set makes induction better and deduction more reliable over longer chains. It doesn't add a third capability that wasn't there before. Feed a model more of the internet and you get a more convincing compositor, not a different kind of thing.

I said this to a commenter under my original bot post, before I'd read the DeepMind paper: a model trained on Newtonian physics at sufficient scale would produce better Newtonian predictions, not special relativity. Turns out that's the whole thesis, just arrived at from a different direction — one from watching my own retrieval logs, one from a formal read of Peirce.

Which brings me back to my own bookmarks.

What actually happened at bookmark-time, every time I decided a tweet was worth saving, was a small version of the jump. This connects to that. This contradicts what I believed last year. Keep it. I've been doing that since 2016 — a decade of small jumps, one save at a time. Bookmark Brain inherited the residue of all of them. It never makes one itself.

That's the part that should worry people more than the benchmark charts do. Not that the model can't out-think Einstein. That most of what gets paid for isn't the jump either.

I write the essay, but I bookmark the argument first. That's where I'm putting the hours now — not in the composing, which the model will keep getting better at, but in the deciding what's worth saving. The jump doesn't scale. Mine, at least, still has to happen one bookmark at a time.

I Was Filming a Demo of My Monitoring Tool. The Monitor Wasn't Monitoring.

Daniel Nwaneri — Mon, 27 Jul 2026 08:40:19 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The project

workers-monitor is a Cloudflare Worker I run to watch a small fleet of
my own Workers. Hourly cron, deterministic threshold gate, Claude Haiku
judgement only if the gate trips, Telegram alert if Haiku confirms
something's actually wrong. A quiet hour makes zero LLM calls.

github.com/dannwaneri/workers-monitor

It pages me on Telegram when something's wrong with the fleet it
watches. What it didn't have, until recently, was an equivalent safety
net for itself — if workers-monitor broke, nobody got told. So I added
Sentry: automatic error capture, and Sentry.withMonitor() around the
hourly run so a dead cron trigger would be caught immediately instead of
silently waiting up to 24 hours for the next daily heartbeat to go quiet.

I deployed it. A cron tick ran clean. I moved on, fairly pleased with
myself.

The bug

A few days later I sat down to record a demo video of the Sentry
integration — screen capture, narration, the whole thing, for a separate
part of this challenge. Simple plan: open the Sentry dashboard, show the
captured errors, show the cron monitor's check-in, done.

I opened the Monitors page to get the screenshot.

There was nothing there. Just Sentry's own auto-created generic "Error
Monitor" — no workers-monitor-hourly-poll, no Cron-type entry
whatsoever, despite the code having run successfully, every hour, for
days. I'd been carrying around a completely false belief: that deploying
Sentry.withMonitor() and watching it execute without error meant it was
monitoring. It wasn't. It never had been.

Sentry.withMonitor("workers-monitor-hourly-poll", () => run(event, env))

Nothing about this throws. It compiles. It deploys. The wrapped function
runs exactly as intended, every hour, on schedule. And it still wasn't
doing the one thing it existed to do.

How I found it

By accident, honestly. Not through review, not through testing — I found
it because I was trying to film proof that something worked, and the
proof wasn't there. If I hadn't been making a video, I might not have
looked at that specific dashboard page for weeks.

The fix

withMonitor's check-in has nowhere to attach without a schedule —
Sentry needs to know what "on time" even means for this monitor before it
will create a Cron Monitor entity to check in against. Without that
config, the call silently has no monitor to report to.

Sentry.withMonitor(
  "workers-monitor-hourly-poll",
  () => run(event, env),
  {
    schedule: { type: "crontab", value: "0 * * * *" }, // matches the real cron trigger
    checkinMargin: 5,
    maxRuntime: 5,
    timezone: "UTC",
  },
)

One config object. I deployed it, then made myself actually wait for a
real hourly tick — not a local test, the genuine production cron —
before I let myself believe it was fixed. workers-monitor-hourly-poll
showed up as a real Cron monitor afterward, "Every hour" schedule and
all.

Why this one mattered to fix

Because it's the same lie twice, if I'm honest about it. I built a
monitoring tool because I don't trust systems to tell me the truth about
their own state unprompted. Then I shipped a piece of that exact tool
without applying the same skepticism to itself. "It ran without throwing"
is not evidence of "it's doing its job" — I know that, I'd have said it
confidently to anyone who asked — and I still fell for the gap between
those two claims on my own code, in the one project whose entire purpose
is not falling for that gap.

See it

https://youtu.be/dSvSOw0fy-E

Real Sentry dashboard, the actual moment the cron monitor showed up after
the fix — not a re-enactment.

What I'd do differently next time

Treat "I deployed it and it didn't error" as a hypothesis, not a
conclusion — for every feature, not just the ones I already suspect are
fragile. The KV read bug I fixed earlier in this project (a separate
entry, if you're comparing notes) came from a structured spec review.
This one came from dumb luck — I happened to need a screenshot. I'd
rather it come from the habit than the accident next time.

PR

github.com/dannwaneri/workers-monitor/pull/3

One file changed, 14 insertions, 1 deletion — the isolated fix, nothing
else riding along with it.