DEV Community: Daniel Nwaneri

I Told a PR Nothing Would Break. Qodo Checked.

Daniel Nwaneri — Tue, 22 Sep 2026 11:51:43 +0000

Every PR description is a claim. "No consumers of this API were changed." "Safe to merge." Most review tools take that claim at face value, because checking it would mean looking outside the diff.

I wanted to know what happens when a review tool doesn't take the claim at face value. So I built two connected services. I made a change that looked completely safe in isolation. I wrote a PR description that explicitly said nothing downstream would break. Then I let Qodo decide whether to believe me.

It didn't.

the setup

Two small repos, deliberately simple:

orders-api — a Cloudflare Worker. GET /orders/:id returns { id, status, total, currency }.
orders-client — a consumer that fetches an order and fulfills it if order.status === 'paid'.

Baseline works end to end. Fetch a paid order, it fulfills. Fetch a pending order, it skips. Nothing exotic.

Then I opened a PR on orders-api that changes status from a plain string to a nested object: { value, updatedAt }. The reasoning in the PR description was real: I wanted an audit trail for when an order's status last changed. The description also said this explicitly:

"No consumers of this API were changed as part of this PR — response shape update only."

That sentence is true and misleading at the same time. No consumer's code changed. But orders-client still does order.status === 'paid', and that comparison (a string against an object) silently returns false forever. Paid orders stop being fulfilled. No crash, no error, no log line that looks wrong. Just orders that quietly never ship.

orders-api's own type checker sees none of this. Nothing in that repo references orders-client. The break is real. It's completely invisible from inside the repo where it originates.

running qodo

I commented /agentic_review on the PR. Ninety seconds later:

Action required: Existing clients misread order statuses 🐞 Bug · Correctness

"Order.status changes from the existing string union to an OrderStatus object while the same route continues serializing the order without versioning or negotiation. Every successful order request now returns an incompatible field to existing consumers, including the documented orders client that this PR explicitly leaves unchanged."

That last clause is quoting my own PR description back at me, the part where I said nothing downstream was affected, and pointing out that it's wrong. Qodo didn't guess this. It checked orders-client, the actual repo, and found the line that breaks.

https://youtu.be/xSrXgXC-Qig

proving it wasn't a lucky guess

One finding could just mean Qodo read my PR description carefully and got suspicious of the wording, not that it checked the other repo at all. I wanted to rule that out.

So I ran a control. Same exact code change. Same PR description, word for word. The only variable: I removed orders-client from Qodo's connected repositories first.

The result, on an otherwise identical PR:

Great, no issues found!
"Qodo reviewed your code and found no material issues that require review."

Connected: one action-required bug, correctly named, correctly explained. Disconnected: a clean pass. Same code, same description, same claim about consumers being unaffected. The only thing that changed was whether Qodo could see the repo the claim was about.

That rules out coincidence. The connection is causal.

what surprised me

The "no issues found" screen is the one that stuck with me, not the bug it missed.

It's polished. Little Qodo mascot, clean green checkmark. If you saw that screen in isolation, it reads like an endorsement: ship it, you're clear. It isn't lying, exactly. Qodo found no issues in what it could see. But "no issues found" and "no issues exist" are different claims, and the UI doesn't visually distinguish between them.

Every review tool has this problem, not just Qodo: a clean bill of health only means as much as the tool's field of view. A confident, cheerful "all clear" from a tool that can't see half the picture is more dangerous than an honest "I don't know," because nothing about the interaction tells you to doubt it.

The fix is exactly what connecting orders-client did: widen what the tool can see before it renders a verdict at all.

single-repo vs. connected

orders-api's own review pipeline (tests, types, linting) has no way to catch this. It was never going to. The bug doesn't live in orders-api. It lives in the gap between two repos that don't know about each other.

This is why cross-repo context matters here: it catches a false claim of safety that a single repo has no way to verify. Most bugs in most PRs are contained to the diff. This one specifically wasn't, and the failure mode when you get it wrong is the quiet kind: nothing breaks loudly, revenue just doesn't fulfill, and someone eventually finds out from a support ticket, not a stack trace.

Single-repo tools aren't wrong to focus on the diff. They're just answering a narrower question than the one that mattered here.

takeaway

Your PR description is a claim, not a fact. "Nothing downstream changed" is something you believe until the thing checking your PR can actually see downstream.

I wrote a true sentence that was still wrong. Qodo caught the wrong part. Then I proved it wasn't luck by taking the same claim, removing the tool's ability to check it, and watching it get fooled by the exact same sentence.

If your review tool can't see the repos your change touches, it isn't reviewing your change. It's reviewing your diff and trusting your description for everything else.

Repos: orders-api · orders-client. Qodo runs on the free tier for public repos — qodo.ai.

My AI Agent Isn't Allowed to Decide Anything

Daniel Nwaneri — Mon, 21 Sep 2026 12:35:25 +0000

It's a few hours before a delivery deadline and the work has piled up. Somewhere in that pile is a piece of work that's about to miss its deadline, unless someone pushes a different piece back to make room. That's the exact moment I built this for.

Three out of five. That's how often Shot-Delivery Guardian, the system I built for Google's Agentic Cinema hackathon, picks the right answer when it's the one making that call, in real tests against the actual running system, not just on my laptop. But here's the number that matters more: zero broken rules. It never once picked a piece of work the client had already signed off on. It never once picked something other work was waiting on. It never once touched anything the director had flagged as important.

The AI never got an important decision wrong, because I never let it make one.

Picture a movie or TV show going through its last steps: editing, effects, color, quality checks, then handoff to the streaming service. If one of those steps backs up, something is now going to miss its deadline unless a person delays something else to make room. That choice has to happen fast, and it can't touch anything the client already approved, can't hold up something else that's waiting on it, can't be the one thing the director cared about most. Right now, a person does this by hand, checking a bunch of screens at once, usually in a hurry.

So I built a system that watches the work as it moves through each step, notices when a deadline is about to be missed, figures out why, and says which piece of work is safe to push back.

Here's the part I care about most: an AI can be great at figuring out what's going on. It shouldn't be the thing that makes the final call on something with real money or a real contract riding on it.

It's tempting to just hand the AI everything: here's what's happening, tell me what to delay, go. I didn't do that. The actual choice comes from a short, fixed list of rules, written as plain code, not from the AI's own judgment:

def score_bump_candidates(shots):
    eligible = [
        s for s in shots
        if not s["client_approved"]
        and not s["director_flagged"]
        and not has_downstream_dependents(s, shots)
    ]
    return sorted(eligible, key=lambda s: slack_hours(s), reverse=True)

In plain words: it throws out anything the client already signed off on, anything the director cared about, and anything other work is waiting on, then picks whichever of what's left has the most spare time before its own deadline. The AI runs this rule and reports the answer back, word for word. It doesn't get to argue with it or dress it up to sound more sure of itself than it is.

I split the system into eight small pieces, one each for intake, rendering, color, quality checks, delivery, scheduling, a test generator, and the AI itself, instead of one giant program, so a slowdown in one place can't take down everything else with it. Between them sits something like a waiting line, because work moves at different speeds in different places, and a slow spot shouldn't block everything behind it.

Every one of those eight pieces reports what it's doing the same way, so I didn't have to build a different reporting system for every tool watching it.

I used three different kinds of monitoring, because one wasn't enough. One tells me how bad the backlog is overall. Another tells me which specific pieces of work are involved and what I know about them, like who approved it and when it's due. A third tells me exactly where, inside one piece of work's journey, the time got lost. No single one answers the whole question.

To let the AI ask all three of these tools questions, and write its answer back onto the same dashboard a person already watches, I used one shared connector instead of building three separate ones by hand. It also logs in on its own, with no person needed to click anything, so the whole thing can kick off automatically the moment something looks wrong.

The AI is doing the part it's actually good at. If it's not making the final call, why have it at all? Turning "the queue is packed and barely clearing" into something a person can act on in five seconds, that's explaining, not deciding. It's also doing the legwork: check the first tool, then the second, then the third, run the rule, write it up. A plain script could do that same sequence in order. It just couldn't explain itself in plain English while doing it.

The answer shows up as a note right on the dashboard the person on call is already looking at. No new place to check.

The cleanest demo isn't always the most convincing one. During one real test, the system hit a real problem reaching one of its three tools halfway through. It said so, plainly, in its own words. It didn't hide the gap, and it didn't guess. It used what it could still get from the other tools, and still gave one clear answer: this exact piece of work, with about an hour of spare time before its deadline, safe to push back.

That's a better proof than a clean run would've been. The fixed rule didn't need the AI to get everything right. It just needed whatever real information made it through.

So, the honest numbers: 3 of 5 real tests matched what the fixed rule alone would've picked. About 44 seconds on average to look into a problem and respond, timed against the real live system. Zero broken rules across every single test.

60% isn't a number I'd normally want to lead with. But it's real, it came from a live system doing real work, and I didn't round it up. I'd rather show an honest 60% than a demo that only ever shows the one time it worked.

Let the AI look into things, explain them, and handle the steps. Keep the piece that actually decides, the one touching money or contracts, as a fixed rule the AI can't talk its way around. It doesn't need to be trusted with the decision to be useful. It just needs to be trusted to explain one.

Code's at github.com/dannwaneri/shot-delivery-guardian, MIT licensed.

ngl, "asked an agent to find where my space went and the agent was the problem" is the most 2026 sentence I've read this week.

Daniel Nwaneri — Fri, 18 Sep 2026 22:36:37 +0000

[Boost]

Daniel Nwaneri — Fri, 18 Sep 2026 08:45:13 +0000

Daniel Nwaneri

Sep 15

1 in 10 Sites Have a Broken llms.txt. Reddit Included.

#ai #seo #python #opensource

5 min read

[Boost]

Daniel Nwaneri — Thu, 17 Sep 2026 16:05:13 +0000

Daniel Nwaneri

Sep 14

AI Made Hackathons Easier to Finish. That's the Problem.

#discuss #ai #hackathon #productivity

3 min read

[Boost]

Daniel Nwaneri — Thu, 17 Sep 2026 07:01:35 +0000

Daniel Nwaneri

Sep 15

1 in 10 Sites Have a Broken llms.txt. Reddit Included.

#ai #seo #python #opensource

5 min read

1 in 10 Sites Have a Broken llms.txt. Reddit Included.

Daniel Nwaneri — Tue, 15 Sep 2026 14:03:28 +0000

Quick recap if you're new here: last month I tested whether five different AI engines (Claude, ChatGPT, Gemini, Perplexity, Bing Copilot) would cite my own websites when answering people's questions. Mostly, they didn't. I chalked some of that up to normal stuff, small sites, not enough content out there about them yet.

Then a reader named riley craig left a comment that changed the story.

He'd checked two files on my sites that I hadn't thought to check myself: robots.txt and llms.txt. Both are supposed to be small text files that live at the root of a website (yoursite.com/robots.txt) and tell automated visitors, like search engines and AI crawlers, what they're allowed to look at and where to find the important stuff. robots.txt has been standard since the 1990s. llms.txt is a newer, similar idea, specifically aimed at AI tools, and it's meant to be a plain, boring text file, nothing fancy.

Mine weren't text files. They were my homepage.

Every time anything, human or bot, requested robots.txt or llms.txt on my site, my server handed back the exact same 35KB webpage you'd get by just visiting the homepage, titled "Remote AI Agent Developer." My other site, naija-vpn.com, had the same problem on llms.txt (its robots.txt was fine, that one worked correctly).

Modern websites are often built as what's called a single-page app, or SPA. One HTML page loads, and then JavaScript swaps content in and out as you click around, instead of loading a fresh page every time. To make that work, the hosting platform (mine is Cloudflare Pages) is usually set up to hand you the main app page whenever you ask for a page that doesn't exist as a real file. The assumption is that you're a person who clicked a broken link and should land somewhere useful.

That fallback is helpful for people. It's a problem for robots.txt and llms.txt, because those specifically need to be small, plain text files, not a webpage. My site's build process, the step that turns source code into the files that get uploaded and served, never generated those two files at all. So every request for them just fell through to the homepage fallback. An AI crawler asking "what am I allowed to read here" got back a full HTML page instead of a straight answer.

I fixed both sites. Confirmed the fix works by checking the files myself afterward. That part took an afternoon.

Fixing my own two small sites isn't much of a story on its own. Anyone could have that exact same bug and never notice, because nothing about it throws an error. Your site still loads fine for anyone visiting it normally. It just quietly gives bots the wrong answer.

So I built a small tool to check for this automatically: ai_crawlability.py, a new piece of an open-source SEO toolkit I maintain. Point it at a domain and it requests robots.txt, llms.txt, and sitemap.xml (a third standard file, this one lists all the pages on a site so search engines can find them), then checks whether each one comes back as a real text file or as a webpage in disguise.

If a site simply doesn't have an llms.txt file at all, that's not this bug. Most sites don't have one yet since it's a new convention, and a normal "file not found" response is completely different from what happened to me. The bug I'm looking for is this: the server says "yes, here's your file" (a status code of 200, which just means "success, request completed normally") but hands you a full webpage instead of the plain text you asked for.

The tool can check specific sites you give it directly. It can also do something more interesting: pull a real, unbiased sample of domains straight from Google search results (using SearchApi, the search API company sponsoring this series of articles) for whatever search terms you give it, then check every domain in that sample. That second mode is the point. Checking my own two sites tells you nothing about how common this bug really is. Checking a real sample does.

What a real sample turned up

I picked ten completely unrelated search topics on purpose, developer tools, travel booking, productivity software, laptop reviews, coffee subscriptions, specifically so the sample wouldn't just turn into a pile of other companies similar to mine. SearchApi's search results gave me 50 unique domains across all ten searches. I checked every one.

Result	Count
Working correctly	39
Has the bug (webpage instead of text file)	5
Blocked the request or errored out (a different, unrelated problem)	6

Five broken out of fifty checked. That's one in ten.

reddit.com has the bug on both llms.txt and sitemap.xml. So does skyscanner.com. paymoapp.com has it on llms.txt. And two different Medium.com blogs, run by two different people (ilampadmanabhan.medium.com and navanathjadhav.medium.com), both have it on sitemap.xml, with the broken webpage they each got back weighing in at 41,975 and 41,972 bytes respectively.

Two unrelated writers didn't independently make the identical mistake. Those numbers being that close means it's almost certainly Medium's own platform serving the same broken response to everyone who points a custom domain at a Medium blog. If that's true, it's not affecting two people. It's affecting every writer who's done that.

Not every failure was this bug, though. Six domains failed outright: w3schools.com, pcmag.com, united.com, cheapoair.com, drinktrade.com, and beanbox.com. Before trusting my own tool's judgment on those, I checked two of them by hand. w3schools.com sends back a "403 Forbidden," which means it's actively refusing the request. pcmag.com does something similar behind what looks like bot-detection software. Neither of those is the bug I'm measuring. They're sites that block anything that doesn't look like a real browser, which is a completely different, unrelated situation, and my tool correctly recognized that and left them out of the "broken" count instead of lumping them in.

That distinction turned out to matter more than I expected going in. A tool that can't tell "this site is blocking me" apart from "this site is actually broken" would have either made the 10% number look bigger than it really is, or hidden real bugs inside a pile of unrelated failures.

I haven't gone back and re-run the original 5-engine test on my own two sites since fixing them. Search engines and AI crawlers don't revisit a site the moment you change something, they come back on their own schedule, which can take days or weeks. Testing again too soon would just show the same zero-citation results as before, but for a completely different reason than last time, and that would be a misleading comparison, not a real one. That follow-up test is its own piece, once enough time has gone by for it to mean something.

For now, here's what I can say with confidence: this bug is real, it's not rare, and reddit.com has it right now.

Where to look

The code: github.com/dannwaneri/seo-agent, specifically modules/ai_crawlability.py
The tool behind the domain discovery: SearchApi's Google search engine

*This piece was produced as part of SearchApi's Developer Ambassador program. They provided API credits; I built and tested the integration myself.

AI Made Hackathons Easier to Finish. That's the Problem.

Daniel Nwaneri — Mon, 14 Sep 2026 14:07:49 +0000

Finishing a hackathon project used to mean something. Now it just means you had a good prompt.

Someone on my timeline said the golden age of hackathons is dead. 2021 to 2023, they said, was the easiest way to build a network, learn fast, and land a job without touching Leetcode. Then AI nuked it. Building got so easy that the "idea guys" took over, and nobody's sure what replaces hackathons for someone starting out today.

I don't fully agree. But I get exactly what they're mourning.

Back before AI tools were part of my workflow, I entered a few hackathons, some Google ones among them, and honestly my memory of most of it is pretty vague now. What I do remember clearly is not finishing. Not because the idea was bad. I was stuck in tutorial hell, dealing with imposter syndrome on top of it.

I still call those hackathons rewarding. Not because I shipped something, most of the time I didn't, but because they showed me what was actually obtainable in this industry. Real working software, built by people my age, solving real problems in a weekend. Before that exposure, "good" was an abstraction. After it, "good" had a shape.

Right now I'm juggling a handful of hackathons at once. Agents for Humans, AI Builders, the Niger Delta VES fleet build, a few others. The biggest difference from those early days isn't the tech. It's that I actually finish now.

I get bored of most games after an hour. Building doesn't do that to me. Endless progress. Rewarding. Mine when it's done. People don't always get that building is the relaxing thing, not the job.

Which is exactly why finishing matters more now, not less. If building is the game, unfinished hackathons are just levels I never beat. At this point somebody needs to electrocute me if I so much as glance at a new project before these are done.

AI didn't fix tutorial hell. It just made it optional. The blocker isn't syntax anymore, or four hours lost debugging an import error at 2am. That friction is mostly gone.

But AI enhances more than speed. It enhances your ability to build the wrong thing, confidently, on schedule, with a working demo to show for it.

That's basically what the "idea guys taking over" complaint is pointing at. Finishing used to be proof you understood what you built, because finishing was hard. Now finishing is available to anyone with a good prompt and no real judgment about whether the thing being built is worth building.

Tutorial hell was miserable. I'm not nostalgic for it. But debugging your own broken import statement at 2am taught you what actually breaks and why. A working AI-generated demo doesn't.

I don't think the answer is bringing that friction back on purpose. I don't think the industry has figured out what replaces it either. AI removed the thing that used to force people to learn slowly. Nobody's built the thing that forces people to learn fast without skipping the part where you actually understand what you shipped.

Hackathons still do the thing that mattered most for me. They show you what's obtainable, who's building it, what the bar actually looks like up close. That part isn't dead. I'm getting it right now, across every one of these I'm juggling.

What's dead is the assumption that finishing proves anything. Finishing used to mean something. Now it just means you had a good prompt.

I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost.

Daniel Nwaneri — Mon, 14 Sep 2026 12:47:08 +0000

I built StacksNG for the Africa Deep Tech Challenge 2026: an offline coding assistant grounded in the docs of four Nigerian fintech APIs: Paystack, Flutterwave, Monnify, Termii. It runs on retrieval, not memory (RAG, if you know the term): pull the real doc passage before answering, cite the source. 780 scraped chunks, running entirely on-device via llama.cpp, the engine that runs the model locally. No cloud, no API keys, no hallucinated endpoints in payments code.

I didn't make the semifinals.

No feedback came with that. So I went looking myself.

StacksNG shipped qwen2.5-coder-7b, stock, compressed down to fit in memory (Q4_K_M quantization). That's the largest model that fits the challenge's 8GB envelope, and it's a deliberate bet against the scoring formula, not an oversight. I wrote the ablation into the report myself. A smaller, 1.5B version of the same pipeline wins the formula by about 35 points: faster, lighter, better on the speed and memory-efficiency scores. I rejected it anyway, because on one of my two registered test prompts, the 1.5B opened with:

import flutterwave
client = flutterwave.Client(...)

That library doesn't exist. The 7B, same retrieval context, wrote real requests calls against real documented endpoints. I picked correctness over the formula and said so in the report. Fine engineering decision. Turned out to be the wrong thing to be defending.

I cloned all 20 published semifinalists (full git history, in case anyone privatizes their repo after judging) and read every technical report. SME-Ledger, Jamii Afya, TaxSabi, CodeFellow, ARIS, Homa, all of them: LoRA, QLoRA, distillation, or a merge. Every single one touched the base model somehow.

StacksNG was the only stock-model architecture in the batch. I noticed the pattern, filed it under "interesting," and moved on. That was the first mistake.

My own submission.json:

"accuracy": []

Empty array. Not a low number. No number. git log on that file shows exactly one commit, never touched again before the deadline. The adtc-profiler run that produced it either skipped the accuracy gate or lm_eval wasn't installed at the time. Same failure mode either way: silent, not an error.

Sacc is 50% of the scoring formula. I'd shipped half the score as zero without meaning to.

Trying to fix that, I installed lm-eval-harness and pointed it at my own model through a local llama-server. It crashed. Every one of 200 requests came back "Invalid logprobs data."

Two bugs, both confirmed by reading the actual source:

adtc_profiler's own accuracy.py calls lm_eval with base_url=local. Not a URL. lm-eval's GGUF backend needs http://host:port and POSTs to {base_url}/v1/completions. As shipped, that line fails for anyone who runs it literally.
lm-eval-harness's GGUF backend expects echo=true to return logprobs for the whole echoed prompt, the old OpenAI completions behavior. Current llama-server only returns logprobs for newly generated tokens. The harness hasn't caught up.

I patched both locally. Instead of trusting the broken echo behavior, I forced the model to generate the exact answer text (a grammar rule constraining what it's allowed to output) and read its confidence scores off that generation directly. Ran the hackathon's own default benchmark, a standard general-knowledge multiple-choice test. Got a real number:

arc_easy acc_norm = 0.74

Competitive. Ahead of one fine-tuned semifinalist, just behind two others. Plugged into my own formula: Stotal goes from the submitted 10.23 to 47.23. A 4.6x swing from one missing number, no architecture change.

I thought that was the end of it.

Your submission's originality score did not meet the threshold required to advance.

Originality Score (0–10): 3

Model Origin (Originality Review): Stock model, used as-is: Qwen2.5-Coder-7B-Instruct-GGUF from lmstudio-community's official quantization.

Round 1 runs an originality gate before any technical scoring even happens. Template compliance: fine. The RAG architecture, the citation grounding, the 780-chunk corpus: not mentioned. The problem was never Sacc. The problem was that I never touched the model.

All the archaeology (the broken base_url, the echo/logprobs mismatch, the 0.74) was real, reproducible, and completely beside the point. I'd spent a week debugging the wrong layer.

The pattern was sitting in my own data the whole time. Eighteen other teams fine-tuned. I read that fact, found it interesting, and filed it as a scoring-formula tradeoff instead of what it actually was: everyone else had cleared a bar I didn't know existed.

That's the uncomfortable part. The evidence was already in front of me before the rejection email arrived. I reasoned right past it anyway.

I was looking at the parts of the system I could inspect: the formula, the profiler source, the tooling. The part I couldn't see was the one that mattered: a review that runs above all of it, undocumented in any code or rule I had access to.

A RAG layer grounding a stock model is a defensible product decision. It is not, to an originality reviewer, evidence that you built anything. Those are different bars, and clearing one says nothing about the other.

If you're entering a hackathon with a scoring formula published anywhere: read it. Optimize against it. But ask what isn't in the formula too. A judging process can have a gate that never shows up in the rubric or the tooling. The only way to find it is to assume it exists and ship something that survives it anyway.

Ship a real modification. Even a light LoRA pass on your own corpus would have cleared this. The application layer was good work: retrieval, citations, the deterministic-gate bugs I fixed after a DEV.to reader found them in production. None of it was ever going to be evidence that the model itself was mine.

Good luck to the eighteen who advanced. You did the part I skipped.

StacksNG: github.com/dannwaneri/stacksng. Two tooling bugs found during this postmortem, filed upstream: adtc-profiler#4 and lm-evaluation-harness#4158.

I Hid a Rule in CLAUDE.md. Only One Reviewer Could Prove It Read It.

Daniel Nwaneri — Wed, 09 Sep 2026 14:40:04 +0000

"Does your AI code reviewer read CLAUDE.md?" is the wrong question to ask. Almost any of them will say yes, and honestly, most will catch something. What actually matters is whether you can tell the difference between a tool that read your rule and a tool that just happens to agree with it.

So I built a small API, wrote real rules into CLAUDE.md and AGENTS.md, and then broke one of those rules twice on purpose. The first break was something any decent security scanner would flag anyway. The second one had no reason to get caught unless a tool actually opened the file and read it. I ran both through CodeRabbit and Qodo to see what would happen.

Both tools caught the second one eventually. Only one of them could tell me why.

the setup

rules-demo-api is a small Cloudflare Worker. POST /feedback takes a rating and a message, validates them, and logs the rating. Both CLAUDE.md and AGENTS.md document the same four rules for the repo, including this one:

"Never log request headers or bodies. They may contain secrets, tokens, or PII."

and one I added later:

"All new route paths must be kebab-case (/feedback-summary), never camelCase (/feedbackSummary). This is an internal convention, not a general best practice."

I wrote that second rule that way on purpose. There's nothing about kebab-case routes that's a security issue, and nothing about it is a bug either — it's just a house style choice with zero backing outside this one repo. Which is exactly what makes it a cleaner test than the first rule. If a tool flags a violation of this one, it had to have read the file, because there's nothing else it could be catching. It's a small example, but this is exactly the kind of gap that shows up once you're running AI code review at scale instead of just eyeballing one repo yourself.

test one: the rule with a built-in excuse

I opened a PR that logs the full request headers and body, with a commit message saying it was "temporary, to debug a client issue." Both tools caught it right away.

Here's what Qodo said:

Rule violation, High: Customer secrets enter application logs
"Rule 1: Do Not Log Unredacted Request Headers or Bodies" (AGENTS.md[5-8])

And CodeRabbit:

"This change can expose credentials and private feedback content through application logs. It should not merge until logging is restricted to an allowlist of necessary diagnostic fields." Merge Risk: High.

Both correct, and only one of them names the actual file. But this test doesn't really prove anything about file-reading either way: logging raw headers and bodies is a well-known bad practice whether or not you've written it down anywhere. Any half-decent scanner flags this on instinct alone. Call this round a wash.

test two: the rule with no excuse

Same repo, new PR. This time I added a GET /feedbackSummary endpoint in camelCase, breaking the kebab-case rule and nothing else. No security issue, no bug, just a naming convention nobody outside this repo has ever heard of. The only way to catch it is to have actually read CLAUDE.md or AGENTS.md.

Qodo caught it immediately:

Rule violation, High: Clients cannot use the kebab-case route
"Rule 4: Use Kebab-Case for New API Route Paths" (AGENTS.md[7-10], CLAUDE.md[7-10])
"Requests to /feedback-summary... do not match the condition and fall through to the handler's 404 response."

It even took the reasoning a step further than I expected. Since the handler 404s on anything unmatched, Qodo worked out that a client calling the correct kebab-case URL would get a 404, while my wrong, camelCase URL worked fine. That's not just matching my code against a style guide. That's tracing what the naming mismatch does once real requests hit it.

CodeRabbit, running on its default "Chill" profile, had a different take:

"No actionable comments were generated in the recent review. 🎉" Merge Risk: Minimal.

Nothing. A clean bill of health on a PR that breaks a rule written into both files in the same repo CodeRabbit had just finished reviewing.

the profile problem

Before I wrote CodeRabbit off, I went and checked its settings. There's a Chill/Assertive toggle in there, and Assertive is documented to surface more style-level feedback. So I switched it over and forced a full re-review on the same PR.

This time, it caught the naming issue:

"Update the route condition... to match the required kebab-case path /feedback-summary instead of /feedbackSummary."

No mention of CLAUDE.md. No mention of AGENTS.md. No rule number, nothing. Just "the required kebab-case path," stated like a fact everyone already knows.

And that's fair, actually, because kebab-case for REST routes really is a common convention across the web, even though I'd framed it here as an arbitrary house rule specific to this repo. So CodeRabbit catching it on Assertive doesn't prove it read my file at all. It's just as easily explained by a stricter style-sensitivity dial reaching for a convention it already knew before it ever saw my repo. I picked a rule I thought had zero generic backing, and I got that part slightly wrong. Better to admit that than pretend the test was cleaner than it was.

what separates them

It's not catch versus miss. Both tools eventually caught the same thing. What's different between them is whether that catch is checkable.

https://youtu.be/_QlBOdiATxU

Every single Qodo finding across both PRs, no matter the severity, points to a specific rule number and a line range in a named file. Each finding also carries its own severity rating, the exact code it's flagging, the reasoning behind the call, and a fix prompt you can hand straight to your own agent. You can click the link yourself. You can go read the rule and confirm it says what Qodo claims it says. I don't have to take any of that on faith.

Update, Sept 10: A reader, pm25coder, pushed back on that last claim in the comments, and correctly. A citation proves a rule was surfaced, not that it drove the decision or that it's still accurate. They proposed a test: move the rule to a different line, change nothing else, and re-run the same PR. If the citation tracks the move, it's a live read. If it doesn't, it's reconstructed from matched text.

I ran it. I shifted the kebab-case rule seven lines down in both AGENTS.md and CLAUDE.md, forced a fresh review on the same PR, and Qodo's new finding still cited the old range: AGENTS.md[7-10], CLAUDE.md[7-10]. Lines 7-10 no longer contain that rule. The citation didn't update.

So the claim above is wrong as stated. You can click the link, but nothing guarantees it still points at the right lines once the file changes underneath it. That's a real gap, not a nitpick, and it's the same failure this whole article is about, just aimed back at itself.

CodeRabbit's findings are real, and often genuinely well-worded, but they never show their source. Whether a given catch came from actually reading AGENTS.md, or from a stricter profile dial turned up one notch, there's just no way to tell from what it hands you. Flipping Chill to Assertive proves the sensitivity is configurable. It doesn't prove the source of any single catch is your file.

At one repo, honestly, that difference barely matters. You'll eyeball the PR either way. But once you're running this across dozens of repos, with rules that get written and updated by different people over time, "did the reviewer actually check the current rule" stops being a question you can shrug off. An unauditable catch and an unauditable miss look exactly the same from the outside. You won't know which one you're getting until it's already gone wrong.

takeaway

Ask a different question than "does it read CLAUDE.md." Ask "can it show me the line it read." One of these tools answers that on every single finding, no exceptions. The other never answers it, no matter how many settings you turn up. That's the dimension actually worth comparing tools on, more than any feature list — see AI code review tools comparison and benchmarks if you want the fuller picture beyond just these two.

A rules file sitting in a repo is best-effort compliance at best: readable by a human if they bother to look, but enforced inconsistently depending on whatever's reviewing it that day and whatever profile happens to be set. A rule that's centrally managed and cited by line number is a different thing entirely. Qodo actually imports rules straight out of AGENTS.md and CLAUDE.md into its own centrally managed Review Standards, where each one picks up a severity and a scope of its own. It's auditable across every repo it touches, whether that's one repo or two hundred of them.

Qodo's own Agentic Toolbox, launching September 9th, pushes this a step further. One of its skills, qodo-get-rules, loads the workspace's centrally-managed rules straight into the agent's session before it writes a single line, not just at review time after the fact. If a rule is worth citing by line number in a PR, it's worth putting in front of the agent before the PR exists.

I hid a rule in a file that most tools claim to read. Getting caught wasn't the hard part. Proving it wasn't a coincidence was.

Repo: rules-demo-api. Both tools run on their respective free/trial tiers for public repos.

My Grandmother Ran Ajo. I Built the Version Where the Pot Can't Walk Away

Daniel Nwaneri — Sun, 06 Sep 2026 22:19:59 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

My late grandmother was a trader. She ran ajo for years, the way a lot of Nigerian market women did, and used her payout round to restock her shop. More stock meant more sales meant more saved for the next round. It worked, until the round it didn't: she told me about a time the person holding the pot borrowed against it to solve a family problem and couldn't pay it back in time. He didn't vanish maliciously. He just couldn't make the group whole again, and the circle absorbed the loss.

That's the actual failure mode. Not fraud, most of the time. Just one person holding money that isn't theirs, under pressure, with no enforcement above their own word.

DEV framed this challenge around ethical and accountable giving, and equity and inclusion. Ajo isn't charity. Nobody donates and walks away. Every member eventually gets their own money back, plus one round early. But it's the original financial-inclusion tool for people the banks never served: market traders, mostly women, building credit and liquidity together because nobody else would. My grandmother didn't have a bank account for this. She had her circle.

Ajo Chain is my answer to that specific story. Before I wrote a line of code, I set one rule: if you deleted a technology, would something actually break, or would the project just look less polished? Everything that failed that test got cut. One whole technology did, and I'll get to why.

Ajo (also called esusu) is a rotating savings circle. Five people, one fixed amount each round, one payout that rotates to a different person every round.

Ajo Chain doesn't digitize the spreadsheet. It replaces the one person everyone has to trust with a Solana program that can't disappear. Escrow, rotation order, and default tracking all live on-chain. Contributions only release to the next member in the fixed order, and only once all five have paid. A missed deadline gets flagged permanently. It's visible forever, not something a moderator can quietly erase later.

The part I didn't expect going in: the smart contract can tell you a contribution didn't arrive. It can't tell you why. So I added one narrow Gemini agent whose only job is reading the on-chain evidence around a disputed default and drafting a plain-language note for a human moderator, distinguishing "the transfer failed on-chain" from "this member shows no attempt to pay." It never finalizes anything. It drafts, a human decides.

Devnet only. No real fund custody.

Demo

Live transparency page, still on devnet, walking a real completed round (four members on time, one deliberate default, Gemini's note, a late catch-up, payout released):

https://ajo-chain-worker.fpl-test.workers.dev/status/3XrTDiMxhznfBZLMxDrEVndP8YKwR1kCaN9HpeUxGRne

Code

https://github.com/dannwaneri/ajo-chain

Program ID: 9RGBLQCcQnsiCdu9RcmEzG4SvihkfTvi1KMz98KCWPoT

How I Built It

Solana is the enforcement layer

This is the only part I'd call load-bearing without qualification. Each group gets an escrow PDA and a fixed rotation order set once at creation. Four instructions: create_group, contribute, mark_default, release_payout. release_payout won't move a lamport until all five members have contributed, and it pays out to exactly one address: whoever is next in the array, checked directly in the handler, not just in an account constraint I could talk myself into trusting.

require!(
    group.has_contributed.iter().all(|&c| c),
    AjoError::NotAllContributed
);
require!(
    ctx.accounts.recipient.key() == group.members[group.round_number as usize],
    AjoError::InvalidRecipient
);

Both checks run before the transfer, not after. There's no recipient field a caller controls; the address is read out of the group's own on-chain state. Delete this layer and you're back to trusting one person with the pot. That's the whole reason the project exists.

I tested it with litesvm instead of a local validator. Three tests, in-process, no faucet, no network: a full happy-path round, a deliberate default followed by a late catch-up that still completes the round, and rejection of non-members and double-contributions.

running 3 tests
test deliberate_default_is_flagged_and_blocks_payout ... ok
test full_round_cycle_pays_correct_rotation_member ... ok
test rejects_non_members_and_double_contribution ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.05s

Full source in programs/ajo_chain/tests/ajo_chain_test.rs if you want to run it yourself: cargo test --manifest-path programs/ajo_chain/Cargo.toml.

Then I deployed to devnet and ran the same story for real, five contributions and one payout, with real signatures and real Explorer links. Somewhere in there I also learned that api.devnet.solana.com blocks Cloudflare's IP ranges outright, which nobody documents until you hit it. QuickNode's free devnet endpoint doesn't have that problem.

Gemini is the fairness layer, and I'm going to undersell it on purpose

It reads the group's on-chain round state plus the disputed member's recent transaction history, then drafts a short note: did the evidence show an attempt that failed, or no attempt at all? That's my grandmother's story again, the same question a human circle asked about the same kind of person, just with transaction signatures instead of memory and reputation. Two different load-bearing questions here, and they get two different answers. For the project as a whole: no, delete Gemini and the Solana program is untouched, nobody loses money, nobody gets locked out. For the dispute-adjudication feature itself: yes. Delete Gemini and a real ajo moderator is staring at raw transaction signatures and err fields, trying to work out whether someone tried and failed or just didn't show up. That's not information a non-technical person can act on. The feature doesn't survive losing it. The product does.

Cloudflare Worker + D1 makes the first two legible to a human

POST /sync indexes a group's on-chain state and its full transaction log into D1. POST /disputes runs the Gemini trigger and stores the result. GET /status/:group is the public page: rotation order, per-round status, the permanent default flag, every transaction linked to Explorer, and the dispute note sitting right there next to it. That page is the actual submission, more than any single instruction is.

What I cut: ElevenLabs

The original plan had a fifth member with no text fallback at all, a voice note as their only interface to the circle. I got as far as an API key, a voice ID, a working TTS call, and then found out ElevenLabs' free tier blocks API access to any premade or library voice. Only a self-cloned voice or a paid plan gets through. I tried Instant Voice Cloning next, got most of the way through the upload flow, and the clone never actually saved. At that point I had a choice: keep spending hours on an integration, or ship a submission that's honest about what it does well. I cut it. The spec file in the repo still has the original design, timestamped.

The bug I almost shipped

Writing an adversarial-test brief for a security pass, I re-read my own /status page code and noticed the dispute note (free-form text Gemini writes) was going straight into HTML with no escaping, on a page anyone can write to since /disputes has no auth. I fixed it, then verified the fix by injecting a real <script> tag directly into the database and confirming it rendered as inert text, not a live tag. That fix is commit 659d6da if you want to see the diff.

Prize Categories

Submitting to:

Best Use of Solana — the escrow, rotation, and default enforcement is the mechanism the whole project exists to provide, not a wrapper around a database
Best Use of Google AI — one narrow Gemini trigger, reasoning over real on-chain evidence, that drafts and never finalizes

Not submitting for ElevenLabs. See above.

Devnet only, no real fund custody, and the on-chain program is the piece I'd stake the submission on. Everything else in this stack exists to make that piece visible and fair to the people actually running the circle.

I Found 3 Security Vulnerabilities in My Own AI Agent's Tool Access

Daniel Nwaneri — Wed, 02 Sep 2026 13:19:03 +0000

I built GeoMart for OpenAI's WebMCP Challenge: a storefront where a human fills in a live, unsubmitted "site brief" and an AI agent uses WebMCP tools to read it, score survey equipment against real physical constraints, and draft a quote the human has to approve. The hackathon's own rule is blunt: every core action has to be reachable only through a WebMCP tool, no REST route allowed to duplicate it.

I thought I'd covered that. Then I asked Google Antigravity to try to break it, gave it the full source code, and it found three problems in about ten minutes.

Antigravity's first pass found that agent-supplied text, the quote reasoning, the product IDs in the comparison tray, was going straight into the page via unescaped innerHTML. It also found something worse: POST /api/quotes, the endpoint that creates a quote request, had zero protection. A bare script from outside any browser could call it directly and submit a quote with no human involved at all.

I fixed both. HTML-escaped every agent-supplied string before it hits the DOM. And added an isTrusted check on the submit button's click handler, so a script-simulated click (element.click(), dispatchEvent) can't fire it. Only a hardware-derived click passes.

I redeployed, felt good about it, and asked for a retest.

Antigravity replayed the exact attack that had worked before. This time it got a 403. Good. Then it tried something I hadn't thought to test: it manually set the Origin header on a bare Node.js fetch() call to match my site's own domain.

It got a 201. The quote submitted. No browser involved.

Origin is not a secret. My repo is open source, required by the hackathon's own rules. Anyone reading worker.ts can see exactly what value my check expects. A real browser can't forge that header, but Node's fetch() isn't a browser and has no such restriction. My "fix" only ever checked "did you bother to copy the domain name," not "are you actually a browser."

If your security check would still pass after you told the attacker exactly how it works, it was never a real check. It was a filter for people who hadn't read your code yet.

I considered a session-cookie approach next and talked myself out of it in about two minutes, because it fails the same test: a script with full HTTP access can just fetch the page first to grab the cookie, then replay it on the real request. No browser needed there either. Same hole, different shape.

The fix that worked needed something that never appears in the repo. I added Cloudflare Turnstile, verified server-side against a secret stored only in a Worker secret, never committed, never in the client bundle. I replayed the exact bypass afterward: correct Origin, no token, 403. Correct Origin, fake token, 403. That's the difference between "a value you can read" and "a value you can't."

While chasing the Origin bypass, the retest surfaced something unrelated: the database table for storing submitted quotes had never been created. I'd written the migration file back when I built the feature and never run it. Every previous test of the "human clicks submit" flow had only exercised the client-side draft step, draft_quote_notes, which never touches the server. The whole point of the human-in-the-loop design is that a human's click writes a row to the database. That write had been silently broken the entire time, and nothing caught it until a click produced a 500 error.

That whole chase left me with one test I now apply to every check I add: does this check depend on something the agent, or anyone reading your public repo, could already know or derive?

Origin headers: public, in your own URL.
Referer headers: same problem.
A session cookie with no server-side validation: derivable in two requests.
A value only your server computes, that never leaves your server: actually a secret.

Adversarial testing an AI-agent-facing app isn't optional if you're claiming a human-approval boundary. Ask an agent with full source access to try to defeat your own claims, specifically the ones you're proudest of, before a judge or an attacker does it for you.

GeoMart is live at https://geomart-webmcp.fpl-test.workers.dev, the WebMCP tools are documented in the repo at https://github.com/dannwaneri/geomart-webmcp, and the commit message on the security-hardening commit walks through every one of these fixes, including the ones that didn't work the first time.