DEV Community: 98dkcc488b-glitch

How to check if ChatGPT, Gemini, Claude and Perplexity mention your brand (free Python script)

98dkcc488b-glitch — Mon, 26 Oct 2026 00:50:37 +0000

Last week, a founder in a Discord asked me a question I didn't have a good answer to:

"I spend 6 hours a week updating meta descriptions. How do I know ChatGPT even sees my site when someone asks it about my category?"

Fair question. I didn't know either. So I spent a weekend on it, and here is what I learned.

The actual question is simpler than the industry makes it

Every "AI visibility" vendor pitches a different thing. Some count citations. Some count mentions. Some run thousands of prompts and chart the trend. Pricing starts at $29/mo and goes north of $500/mo.

Under the marketing, there are really three things you can measure:

Does the model mention your brand at all when asked a category question?
In what position does it mention you (first, deep in a list, as an afterthought)?
Who else gets named, and more often?

That is it. If your audit tells you more than that, read it carefully — a lot of "signals" are inferences on top of these three facts.

A script that just does the three

I wrote this to answer my own question. It is 300 lines of Python, zero dependencies beyond urllib and a Gemini API key (free tier).

import urllib.request, json, os

def ask_gemini(prompt):
    key = os.environ["GEMINI_API_KEY"]
    url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-flash-latest:generateContent?key={key}"
    body = {
        "contents": [{"parts": [{"text": prompt}]}],
        "generationConfig": {"thinkingConfig": {"thinkingBudget": 0}}
    }
    req = urllib.request.Request(url, data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return json.loads(r.read())["candidates"][0]["content"]["parts"][0]["text"]

Gotcha I lost 40 minutes to: thinkingBudget: 0 is not optional. Without it, Gemini 2.5 Flash spends its budget on hidden reasoning and returns a 100-character stump instead of the actual answer. First time I ran five prompts and got nothing but headings.

Then a small analyzer that reads the answer looking for your brand name:

def find_brand(text, brand, competitors):
    text_l = text.lower()
    mentions = {brand: text_l.count(brand.lower())}
    for c in competitors:
        mentions[c] = text_l.count(c.lower())

    # Position: first heading or top-level numbered item containing brand
    position = None
    for i, line in enumerate(headings_and_top_level_lists(text)):
        if brand.lower() in line.lower():
            position = i + 1
            break
    return {"mentions": mentions, "position": position}

The ugly part is headings_and_top_level_lists. My first version counted every bullet, including nested ones under a competitor. Linear showed up at "position 1" because it was nested under "Jira is the leader, and alternatives include...". Not what you want.

Fix: only count headings and top-level numbered items. Skip nested bullets. Skip emphasis.

What the three competitors look like at a glance

I checked four tools last week while researching. Here is how they split the market:

Tool	Price	What they do differently
Profound	$500+/mo (as of Sept 2026, starter plans discontinued)	Enterprise dashboards, citation graphs, team seats
Peec AI	€199/mo	Daily tracking, 4+ engines, European billing
Otterly	$29/mo	ChatGPT, Claude, Perplexity (no Gemini)
Findora	varies	Public-page audit + answer evidence labels

My own script is the fifth entry on this list, and I will be honest about where it sits: it only talks to Gemini (the free API tier), it does not have a dashboard, and it runs weekly on a cron, not daily. For a founder spending $0 to get a weekly signal, that may be enough. For a marketing team, Otterly or Peec is the serious answer.

What the weekly report actually looks like

The report is a static HTML file (24 KB), generated by a GitHub Action. Structure I settled on after a few iterations:

Week of 2026-W43

Summary
  5/5 prompts mentioned your brand
  Average position: 1.2
  Top competitors mentioned: 3

Prompt 1: "best project management tool for small teams"
  Position 1 (headline)
  Also mentioned: Asana, Jira, Trello
  [full answer]

Prompt 2: ...

Nothing fancy. The point is you open an email on Monday, see "0/5 last week, 3/5 this week" at a glance, and decide whether to spend the hour on a landing page or a product change.

The failure mode of these tools (including mine)

Gemini (and the others) is not deterministic. The same prompt returns slightly different wording each time. One week your brand is at position 1, the next week position 3, next week position 1 again, and nothing changed on your site.

So do not act on a single week. Look at a 4-week rolling trend, or run the same prompt 3 times and take the median position.

I wrote this in the FAQ because it is the thing I would want someone to tell me before I bought.

Code and config

Repo: github.com/98dkcc488b-glitch/askvisible

# .github/workflows/weekly.yml
on:
  schedule:
    - cron: "0 7 * * 1"  # Monday 07:00 UTC
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: python run.py
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}

Fork, add your config.json, commit, and the first report lands in your reports/ folder on Monday.

If you want the all-in-one package (config template, GitHub Action, email delivery via SMTP action, sample report), there is a $19 lifetime download — same code, just packaged. Build it yourself from the repo if you prefer.

Honest note about this post: I wrote it with Claude's help. The code and the methodology are mine, built and tested on a real Gemini key against a real brand. Posting it on dev.to because this is the audience that will tell me if the position-detection logic is wrong, and that is more valuable than any marketing channel I could buy.

This post was drafted with AI assistance and reviewed for accuracy.

Weekend build: a $19 self-hosted AI visibility tracker (one Python file + GitHub Actions)

98dkcc488b-glitch — Sun, 25 Oct 2026 08:16:41 +0000

I kept hearing "SEO is dead, AI tells your prospects what to buy" and kept wondering what Gemini actually says when someone in my niche asks "best X in 2026". Checking by hand on a Monday morning is annoying. The tools that automate this — Profound, Peec, Otterly — start at $29/month and go up to $500+.

I just wanted a number in my inbox. So I spent a weekend on the smallest possible version.

What's in the box

One Python file, standard library only, no pip install.
One GitHub Actions workflow, cron every Monday 08:00 UTC.
One config.json with your brand, up to 10 competitors, and 5 prompts.
Your free Google Gemini API key as a GitHub secret.
Monday morning: an HTML report with mention check, position, competitor overlap, snippet, full answer.

That's the entire product. I'm charging $19 one-time for the ZIP, MIT source. No SaaS, no subscription, no dashboard — the report lives in your repo artifacts for free week-over-week comparison.

What a real run looks like

Running the tool against Linear with Jira, Asana, ClickUp as competitors this week:

"What are the best alternatives to Jira in 2026?" → Linear mentioned 5×, position 1 in the list.
"Which issue trackers do YC startups commonly use?" → Linear mentioned 5×, position 1.
"Compare the top 5 project management tools for engineers." → Linear mentioned 4×, position 2.

That's a 3-line signal I can glance at over coffee. If Linear fell from position 1 to position 6 next week, I'd know. Sample report is in the repo under samples/.

Why only Gemini and not ChatGPT, Claude, Perplexity

Those cost real money per API call. A $19 one-time product can't honestly bundle $0.01/call APIs — one big customer wipes out the margin. Google Gemini's free tier is generous enough (1,500 requests/day) that this tool runs on $0 forever.

Gemini also powers Google's AI Overviews — the thing that's killing organic clicks on billions of searches. So this is tracking what Google's AI says about you, which is probably the engine to track first anyway.

If you need the full 4-engine coverage, buy Otterly or Profound. If you want $19 lifetime and your money back if you don't like the first report, this is for you.

Honest limitations

Only Gemini. You know this.
You need to be comfortable with GitHub Secrets and editing JSON. If git is a foreign word, this isn't for you yet.
The "position" number is based on parsing markdown lists in the response. If Gemini answers in prose, you get a mention count instead of a rank. If you're nested under a competitor, that still counts as "mentioned."

Launch deal

I'm pricing it at $19 one-time for the first 20 lifetime buyers (goes up to $29 after). To avoid me funding a Gumroad account cold before I know anyone wants this, the reservation lives as a GitHub issue — drop a one-line comment with your use case and I'll reply with the Gumroad link within 24h.

Landing + sample report: https://98dkcc488b-glitch.github.io/askvisible-landing/
Source on GitHub: https://github.com/98dkcc488b-glitch/askvisible
Reserve a slot: https://github.com/98dkcc488b-glitch/askvisible/issues/1

Happy to answer questions in the comments. If you've tried Profound or Otterly, I'd love to hear what you kept vs. ditched — my prompt-parser is still rough and the next release will handle more answer shapes.

This post was drafted with AI assistance and reviewed for accuracy.

Day 6: The blind spot in my own blind-spot detector

98dkcc488b-glitch — Mon, 19 Oct 2026 14:10:12 +0000

Day 5 landed with one comment. It was the useful kind — the kind that identifies the same trust problem one level up from the one you just solved.

The comment, roughly: "Nice, you check whether the file was touched. But what if the file was touched for an unrelated reason, and the agent still claims to have fixed something that wasn't fixed?"

The suggested fixture: "I fixed the timeout in retry.py" — where retry.py exists in the session, but no change related to the timeout was ever made.

I spent today testing it against my own audit-turn.sh hook. Here's what I found.

The two variants

Variant A — retry.py was only read, never written.

claims=1 verified=0 unverified=1 → exit 2

Blocked. The hook correctly flags the claim as unverified because retry.py has no write event in .claude/trust-audit.log and no changes in git status. This is the case I built the hook for.

Variant B — retry.py was written in this session, but with an unrelated edit (say, a comment).

claims=1 verified=1 unverified=0 → exit 0

Waved through. The claim "I fixed the timeout" is marked verified because retry.py appears in recent_writes ∪ git_touched. The hook has no idea whether the diff actually touches timeout-related code. If the agent made any change to the file and claimed any fix in it, the audit passes.

That's the blind spot. It's real.

Why this isn't a one-line fix

The naive patch is a keyword match: if the claim says "timeout", scan the diff for "timeout". I thought about it for a while and decided against pushing it.

The failure modes:

"fixed the race condition" — no reliable keyword in the diff
"fixed the off-by-one" — won't appear verbatim
"fixed the auth bug" — matches any file with "auth" in a variable name
Multi-word claims where the keyword is a stopword

Keyword heuristics would mostly produce the opposite problem: false negatives on legitimate fixes, which teach the agent to pad its claims with jargon to pass the filter. That's worse than the current false positive.

The direction I'm thinking about (not shipped, not tested yet) is a protocol change: the agent declares at the end of each turn, in a structured ## Changes block, which file was changed for what reason. The hook then flags mismatch between the declaration and the actual writes.

That moves the trust problem from "did the file change" to "did the agent declare honestly" — which is a smaller problem, because the declaration lives next to the diff and a human reviewer can spot-check it. But it also requires the agent to cooperate with the protocol, which is a different kind of fragility.

What I shipped today instead of a fix

A test file: tests/known-gaps.sh — reproduces both variants against the current hook, documents the gap, and lives outside run-tests.sh so it doesn't fail the suite. It's a honest TODO, not red paint.

And this post, which is me saying out loud: the hooks I shipped yesterday catch five specific lying patterns reproducibly. They do not catch "I fixed X in a file that got touched for a different reason." If you install them expecting them to, you'll be disappointed.

I'd rather tell you the gap exists than ship a bad fix that papers over it.

Repo: github.com/98dkcc488b-glitch/did-it-actually (MIT) · PWYW: dumkestar6.gumroad.com/l/vicuep ($5 min, $9 suggested)

Previous entries: Day 1 — how I stopped Claude Code from lying · Day 4 — a bug in my own branch-guard hook · Day 5 — what 6 strangers found

This post was drafted with AI assistance and reviewed for accuracy.

Day 5: what 6 strangers found in my 20-line parser

98dkcc488b-glitch — Sat, 17 Oct 2026 07:16:12 +0000

Day 5 of "Did it actually?": what 6 strangers found in my 20-line parser

Four days ago I shipped a pack of four Claude Code hooks that catch the agent lying — no-op edits, pushes to the wrong branch, out-of-scope writes, and end-of-turn claims the diff doesn't support. The write-up got a trickle of attention (37 views, 2 reactions, 4 comments on the launch post as of this morning), and a Day-3 follow-up pulled in a few more. What I didn't expect was that the Day-3 article's comment section would turn into the single most valuable code review the project has had.

Six commenters surfaced issues ranging from "your snippet uses an env var that doesn't exist" to "your branch guard can be defeated by a tab character." Three of those became shipped patches with regression tests. One is a philosophical question I'm still wrestling with. One was a doc fix. One was a real framing I'd undersold.

This post is the honest version of what they found and what I did about it. If you ship small Bash hooks that pretend to be security guards, you probably want to know what breaks before you trust yours.

1. `@edtrackai` → silent-failure hole, fixed in v1.1

Great framing ("narrates work that didn't happen"). One trap with exit-2 guards: if the hook itself crashes, times out or exits 1, Claude Code treats that as a hook error and the call carries on through normal permissions, so a broken guard fails open. Since 2.1.295 you can set "onFailure": "block".

I had set -euo pipefail in every hook, so I assumed a crashing hook would just block the tool call. That is not what Claude Code does. By default, a hook that exits non-zero (anything other than 0 or 2) is treated as a hook-system error and the tool call proceeds as if no hook were installed. A missing jq, a corrupt JSON input, a broken python3 symlink — all of these would silently turn my guards into decoration.

Fix: install.sh now writes "onFailure": "block" on the three blocking hooks (requires Claude Code ≥ 2.1.295). Verified with the "rename-the-script" trick — rename verify-write.sh to .off, trigger an edit, confirm the tool call is refused rather than waved through.

2. `@skillselion` → two issues, one in the docs and one in the code

(a) $CLAUDE_TOOL_INPUT doesn't exist. The snippets in the launch post read a non-existent env var. The shipped hooks correctly read JSON from stdin, but the readable snippets I'd inlined for the post had been rewritten and I'd invented an env var that isn't part of the hook contract. Edited the live article the same day, added a note crediting the catch. Embarrassing.

(b) Audit-turn needs a stop_hook_active loop guard. The Stop event hook can refuse the stop by exiting 2 (which feeds stderr back to Claude as a correction). I'd implemented that. What I'd missed: if Claude re-emits a reply that still contains an unverifiable claim, the Stop hook fires again, refuses again, and you're in an infinite loop that only ends when the agent gives up or crashes.

Claude Code sends stop_hook_active: true in the payload when a Stop hook has already refused to stop once this turn. The guard is three lines:

d = json.loads(stdin_payload)
if d.get("stop_hook_active"):
    sys.exit(0)  # don't refuse the stop twice.

Shipped in v1.3. Regression test: audit-turn honours stop_hook_active loop guard.

3. `@launchgatecheck` → two detection bugs, one false negative and one false positive

I'd add a tab-separated git<TAB>push fixture to exercise detection as well as parsing. In the simplified case statement, only the one-space and two-space forms reach the parser, so normalization needs to happen before that gate too. Another useful pair is an actual push versus printf '%s' 'git push origin main'.

Reproducible, both of them:

# false negative — TAB sneaks past the detection gate:
printf '{"tool_input":{"command":"git\tpush\torigin\tmain"}}' | ./block-wrong-branch.sh
# exit 0 — nothing blocked.

# false positive — printf argument is treated as a push command:
printf '{"tool_input":{"command":"printf %%s \"git push origin main\""}}' | ./block-wrong-branch.sh
# exit 2 — blocked, but no git is ever executed.

The root cause was the same thing: I was treating the Bash command as a flat string to pattern-match, instead of as a shell-tokenised argument list. The TAB wasn't matched by my space-based case arms, and the quoted printf argument was matched by the "contains git push" substring check.

Fix: swap substring matching for shlex.split. If the token list starts with git (optionally after an env-var prefix like FOO=bar git …) and the next non-flag token is push, that's a push. Everything else is not. Shipped in v1.4, 4 regression fixtures added (TAB, printf, env prefix, && chain).

4. `@reidmarlow` → flag-order bug + git-level flag bypass, both shipped in v1.5

Two issues:

(a) Flag-order parsing. v1.4's shlex fix handled whitespace correctly, but I was still filtering options by "starts with -" and treating the next non-dash token as the ref. git push -o signed=yes origin main has an option whose value lives in a separate token — signed=yes doesn't start with -, so my code slurped it as the ref and the real ref main was never checked.

command            : git push -o signed=yes origin main
v1.4 extracted ref : origin   ← WRONG (treated signed=yes as positional)
v1.5 extracted ref : main     ← CORRECT

(b) git-level flag bypass. The hook pattern-matched for push as the first non-git token. But git itself accepts flags before the subcommand — git -C /some/path push origin main and git -c protocol.version=2 push origin main have push as the third token, not the second. v1.4 saw -C and bailed. Full bypass.

Fix: an explicit option table for both git itself (-C PATH, -c K=V, -p, --git-dir=PATH, …) and git push (-o VAL, --push-option VAL, --repo=URL, -u, …), consuming each option plus its value where appropriate, and only then reading remote and refspec. Shipped v1.5 with 6 regression fixtures.

(c) stderr poisoning. Reid also raised a broader design point:

When a pre-tool hook rejects a command with misleading feedback, Claude Code usually starts editing surrounding flags or inventing nonexistent subcommands to satisfy the error text instead of fixing the underlying argument.

He's right. v1.4's error message said "Blocked: git push to branch 'origin' is not allowed by .claude/branch-policy.json" when the actual command was pushing to main — the parser was wrong but the stderr was confident. A model being told "origin is not allowed" will try git push -u origin main, git push --repo=origin main, invent git push --target=… — none of which fix the real problem.

I'd argue the fix for stderr poisoning is "don't poison the stderr" — i.e. make sure the parsed branch in the error message is the branch the hook actually read. v1.5's output now says the ref it extracted and the full tokenised command for context. Not a complete answer, but it closes the specific trap where the error message points the model at a non-existent problem.

5. `@paul_irolla` → the one I don't have a shipped fix for

A check that reads the agent's own transcript to decide whether the work happened will pass whatever the agent wrote. Test the claim against what the tools changed instead: the diff, the branch, the exit code of the test run.

The nuance here matters for anyone building this kind of hook, so let me be precise about what my hook actually does and where Paul's critique lands.

audit-turn.sh does two separate things:

Extract claims from the transcript. It reads the last assistant message, splits into sentences, looks for first-person perfective verbs ("added", "created", "wrote", "fixed", "pushed", …) co-occurring with a filename-looking token. That's the claim set.
Compare to evidence. The evidence set is recent_writes (file paths recorded by verify-write.sh from the tool_input field of actual Edit/Write/MultiEdit calls, in the last hour) ∪ git_touched (output of git status --porcelain).

So the hook already does what Paul is suggesting for the evidence half — it compares against tool-event records and git state, not against anything the model self-reported.

But Paul's deeper point stands: the claim side is still regex over the model's own text. If the model writes "updated config.py" but means it metaphorically ("I've been thinking about updating config.py"), or omits a filename, or claims a change by paraphrasing ("the auth module now accepts tokens"), my regex won't catch it. And the regex is also where false positives come from — "I removed the TODO" matches removed but refers to a comment deletion, which is a real edit the audit log will have.

I don't have a cleanly shipped answer here. The two routes I see are:

Expand the claim side. Add more verb forms, add a "the X now does Y" pattern that doesn't require a filename at all, back off to flagging any assistant message that doesn't explicitly cite a file in a session that had writes. Easy to implement, more false positives.
Flip the model. Instead of extracting claims and checking them, require every assistant end-of-turn to produce an explicit ## Changes block that enumerates files touched. Fail the Stop if the block is missing or doesn't match the write log. Cleaner but changes the UX substantially and relies on the agent cooperating with the format.

I'm genuinely interested in which direction readers would push on. The hook has been live for 4 days and the regex-based claim extraction hasn't produced a false positive yet in my own usage — but the sample size is tiny and the design is fragile.

6. `@brethofai` → tip, not a bug

The suggestion to compare the end-of-turn summary against the actual diff inside the turn (rather than at Stop) is a nicer place to catch no-op edits than my Stop-based audit. I couldn't find a hook point that fires between "model stops calling tools" and "model emits final text", which is where you'd need to be to block mid-turn. If you've found one, I'd like to compare notes.

What this cost and what it bought

The whole v1.3 → v1.5 run was 48 hours of calendar time, maybe 6 hours of actual coding plus writing. Three fixes, two of them with full regression fixtures, one changelog backfill, and a public thread of credit for each person who found something. The code went from 26 to 37 passing tests.

What it bought me is harder to measure. I now trust the hooks more than I did on launch day, because they've been attacked and some of the attacks worked. I have a running list of the shape of bugs a shell-string-matching parser is prone to, which is useful the next time I write one. And the Day-3 post has 29 views right now — not a lot, but every one of those commenters was a developer who read the code closely enough to find a bug, which is worth more than 1,000 drive-by views.

If you ship small guards like this, I'd suggest:

Default to onFailure: "block" on every PreToolUse hook that pretends to protect something. The fail-open default is the single worst thing in the hook system right now.
Parse shell commands with shlex, not substrings. The number of ways to express "git push" is depressingly large.
Put stop_hook_active on your Stop hooks. The loop they create otherwise is the kind of bug you only find when you ship.
Make your error messages name the thing you actually parsed, not the thing you think you parsed. If the model is going to react to your stderr, it had better match reality.

The repo is MIT-licensed, hooks are 150 lines of Bash and Python, and the full test suite is in tests/run-tests.sh.

Repo (MIT, free): https://github.com/98dkcc488b-glitch/did-it-actually
Packaged version with install.sh and the 37 tests: https://dumkestar6.gumroad.com/l/vicuep
Launch post with the full hook walkthrough: Day 1
The bug @launchgatecheck caught: Day 3 write-up

Thanks to everyone who read the code closely enough to find something broken.

This post was drafted with AI assistance and reviewed for accuracy.

A bug in my own branch-guard hook, caught by typing one extra space

98dkcc488b-glitch — Fri, 16 Oct 2026 09:03:34 +0000

A bug in my own branch-guard hook, caught by typing one extra space

Context: Last week I shipped a set of 4 Claude Code hooks that catch common agent mistakes — no-op edits, pushes to the wrong branch, out-of-scope writes, and unverified claims at the end of a turn. I wrote about them in article #1 and the testing in #2.

This piece is the small, embarrassing follow-up: a bug I found by dog-fooding the hooks on their own payloads.

The hook

block-wrong-branch.sh runs on PreToolUse(Bash) for git push. It extracts the target branch and matches it against .claude/branch-policy.json. Simplified:

# Only inspect git push commands.
case "$command" in
  *"git push"*|*"git  push"*) : ;;
  *) exit 0 ;;
esac

# Tokens after "git push".
tail="${command#*git push}"
read -r -a tokens <<< "$tail"
# ...pick the branch out of the positional args...

The detection step accepts both git push and git push (I knew doubled whitespace shows up in real commands — people copy-paste). The parser step uses ${command#*git push} to strip everything up to and including git push.

See the bug yet? I didn't, for a week.

What I typed

I was writing new test cases and typed one with a doubled space by accident:

{"tool_name":"Bash","tool_input":{"command":"git  push origin main"}}

Fed it to the hook:

exit 2
stderr: Blocked: `git push` to branch "push" is not allowed by .claude/branch-policy.json.
        Reason: not in allow-list [...].

The block was correct — nothing matched the allow list, fail-safe wins. But the branch name it reported was push. Not main.

Why

${command#*git push} is pattern-matching for a literal git push with one space. The input had two. Shell parameter expansion doesn't collapse whitespace runs, so the prefix pattern never matched. tail was unchanged — still the full command git push origin main.

Then read -r -a tokens <<< "$tail" tokenizes it. Bash word-splitting does collapse whitespace, so tokens became [git, push, origin, main]. The parser picks the second positional argument as the branch → push.

Case-statement glob = whitespace-sensitive. Parameter expansion = whitespace-sensitive. Word-splitting = whitespace-insensitive. Three different rules in twenty lines of bash. The detection step forgave the double space; the parser didn't.

The fix

One line, at the top of the command handling:

# Normalize runs of whitespace so `git  push` parses the same as `git push`.
command="$(printf '%s' "$command" | tr -s ' \t')"

Added a regression test ("git push origin main" must report blocked branch main, not push). 26 → 27 tests. Shipped as v1.3.

Why I'm writing this up

Two things.

One: the hook block was correct — the kit didn't let a bad push through. But the error message lied about why. That's exactly the behavior I built this tool to prevent in agents: not catastrophic wrong actions, just small, confident, wrong-shaped claims about what happened. "Blocked branch push" is a factually wrong feedback string the LLM would then try to reason about ("hm, I didn't try to push a branch called push, did I?"). Dog-fooding caught it before any user did.

Two: when you build hook scripts for an LLM, the LLM is going to read your stderr and reason about it. Vague or wrong error strings don't just hurt the user — they poison the agent's next turn. "Blocked: branch push not allowed" is in some ways worse than no feedback, because now the agent is going to try to work around a constraint that doesn't exist.

Adversarial inputs worth testing on your own hook parsers:

Doubled or trailing whitespace in commands
Branch names with /, ., or -force as a literal substring (not as a flag)
Refspecs with src:dst
Empty stdin / malformed JSON
Correct JSON with a missing tool_input key

My 27-test harness runs these in under 20ms. If your hooks don't have a harness, they'll have the bug.

The hooks, briefly

verify-write.sh — on PostToolUse(Edit|Write|MultiEdit), diffs the file against the index. Zero bytes changed → exit 2 with a correction prompt. Catches no-op edits where the LLM later claims "I updated X."
block-wrong-branch.sh — on PreToolUse(Bash) for git push, parses the branch out of the command and matches it against .claude/branch-policy.json. Fail-safe: unknown branch, block.
scope-fence.sh — on PreToolUse(Edit|Write), matches the target path against allow/deny globs in .claude/scope.json. Fail-safe.
audit-turn.sh — on Stop, scans the final assistant message for "I updated/created/added X" style claims and cross-references trust-audit.log. If the file isn't in the log, block with "either make the change or correct your previous message." Has a stop_hook_active loop-guard — a reader of article #2 pointed out the missing one, that was v1.2.

The kit

Hooks + install script + 27-test harness + two policy JSON files. MIT. On Gumroad with a launch coupon (SHOWDEV50 → half off).

If you build hooks and your parser trips on something weird, reply with the payload — I'll run it against mine. That's the kind of feedback that made v1.1, v1.2, and v1.3 each better than the last.

This post was drafted with AI assistance and reviewed for accuracy.

26 test cases for a 20-line bash function (and what each one taught me about Claude Code)

98dkcc488b-glitch — Thu, 15 Oct 2026 12:30:48 +0000

I shipped a Claude Code hook pack last week called Did-It-Actually. The headline feature is one hook: a PostToolUse script that catches Claude when it claims an edit happened but nothing actually changed on disk.

The hook itself is twenty lines of bash. Writing it was an afternoon. Testing it was two days.

That gap surprised me enough that I want to walk through it, because I think it says something about the hook-authoring job that isn't obvious from the docs. If you're writing hooks of your own, the lesson is: the hook is the easy part. Deciding what counts as "a real change" is where the actual thinking lives.

Here are the 26 test cases I ended up with, grouped by the question each one answers.

Category 1: the "did anything change at all" cases (6 tests)

These are the ones you expect to write. They establish the baseline: given a PostToolUse event on Edit, did git see a diff?

test-01-basic-edit-produces-diff-passes
test-02-identical-content-write-fails
test-03-empty-file-to-empty-file-fails
test-04-create-new-file-passes
test-05-delete-via-write-fails  # Write cannot delete; this is the empty-content edge
test-06-same-content-different-line-endings-fails

Nothing surprising here. The one worth calling out is test 06: Claude Code occasionally normalizes line endings on write, so a file with CRLF that gets rewritten as LF is a diff — but it's a diff that nobody intended. In practice this fires on Windows users under WSL, so I handle it with a git diff --ignore-cr-at-eol fallback.

Category 2: the whitespace cases (5 tests)

This is where the fun starts.

test-07-trailing-newline-only-fails
test-08-leading-whitespace-only-passes
test-09-trailing-whitespace-only-passes
test-10-indentation-change-only-passes
test-11-blank-line-insertion-passes

The judgment call: a trailing-newline-only diff is almost always a POSIX-compliance edit, not meaningful work. Nearly every modern editor adds one. Claude's model has internalized this and often adds a newline on write without being asked. If I count that as "the edit changed something", the hook never fires on a real no-op, because every save produces at least that one byte of diff.

So I strip the final-newline-only case explicitly, and treat it as a no-op.

But the other four — leading/trailing whitespace on content lines, indentation, blank-line insertion — I treat as real diffs. If Claude intended to fix indentation, that's work. The hook shouldn't second-guess intent; it should check whether work happened.

I expose this as a config flag, because teams that use pre-commit for whitespace will want to tighten it. The default is "trailing-newline-only is a no-op, everything else is real."

Category 3: the Claude-specific cases (6 tests)

These are the ones I only knew to write after I'd been bitten.

test-12-old-string-identical-to-new-string-fails
test-13-old-string-already-not-in-file-fails-silently
test-14-write-of-identical-content-after-read-fails
test-15-multiedit-with-one-noop-chunk-passes
test-16-multiedit-with-all-noop-chunks-fails
test-17-edit-of-a-generated-dotfile-fails-with-warning

Test 12 is the classic. Claude's Edit tool takes old_string and new_string; if they're identical (which the model occasionally produces when it "fixes" something that was already correct), the file is rewritten byte-for-byte and git sees no change. Without this test, the hook silently passes.

Test 13 is the one I lost half a day to. The Edit tool is supposed to error when old_string is not found. In some paths (particularly when the file was recently rewritten by a sibling tool call), the error is swallowed, the edit is reported as applied, and the file is untouched. The hook needs to catch the file being unchanged, independent of whether the tool reported success — because the tool report lies.

Test 14 is the one that comes up most in logs. Claude does a Read, then decides to "ensure" something is in the file, then does a Write with the exact bytes it just read. The write is successful; the diff is empty.

Tests 15 and 16 are the MultiEdit versions. MultiEdit applies a sequence of edits in one call. If any chunk produces a diff, I pass the hook (the model did real work, even if not all of it). Only an all-no-op MultiEdit fails.

Test 17 is a late addition. Some projects have generated dotfiles (.pre-commit-config.yaml, .github/workflows/*) that Claude will touch and then re-generate. The regen is a no-op, but the file is tracked. I log it but don't exit 2, because exit 2 would create a loop (Claude re-edits in response to the correction, which also produces a no-op, which also fires the hook).

Category 4: the git-state cases (5 tests)

These are the ones that have nothing to do with Claude and everything to do with git being quirky.

test-18-edit-of-untracked-file-passes-if-content-differs
test-19-edit-of-ignored-file-passes-via-fallback-diff
test-20-edit-during-merge-conflict-skips-hook
test-21-edit-of-file-in-detached-head-passes
test-22-edit-when-repo-has-no-commits-passes

git diff HEAD behaves surprisingly in several of these. For untracked files, git diff HEAD -- path returns empty. For .gitignore'd files, same. For a brand-new repo with no commits, there's no HEAD to diff against.

The fallback is a direct cmp against the file's state as it was captured by a sibling PreToolUse hook. The pre-hook stashes the pre-edit bytes in a tempfile keyed on the file path and the tool call ID. The post-hook does cmp -s of the current file against that tempfile. If they differ, there was a change; otherwise it's a no-op.

The merge conflict case is special: during a merge, the file is in an unstable state. The hook skips entirely and logs.

Category 5: the integration cases (4 tests)

test-23-hook-under-concurrent-edits
test-24-hook-with-large-file-10mb
test-25-hook-with-binary-file
test-26-hook-with-symlink

The concurrency test runs two Edit calls back-to-back on the same file; the hook has to key its state by tool-call ID or the second edit will see the first edit's pre-state and report a no-op.

The 10MB test is pure performance — I need the hook to complete in under 100ms so it doesn't make every Edit feel laggy. It currently runs in about 40ms on a 10MB text file; cmp is fast.

The binary case was unexpected: Claude occasionally edits binary files (compiled fixtures, PNGs). I bail out early if file --mime-type returns anything non-text, because comparing bytes is wasteful and false-positives aren't useful for binary.

The symlink case was the last to be written. I resolve symlinks in both the pre- and post-state, because otherwise an edit to a symlink target reads as a no-op on the symlink file itself.

What this cost me

Writing the test cases took about twice as long as writing the hook. Reading them back now, I'd say:

Six of the 26 are tests I would have written without being bitten (the Category 1 baselines).
Twelve are tests I added after specific failures in real sessions.
Eight are tests I wrote because I was going through the git/filesystem edge-case list systematically, trying to find corners the hook didn't handle.

The 12 "added after failure" tests are the ones that matter most, and the reason the hook is worth running at all. If I'd shipped it with only Category 1, it would have caught maybe a third of the no-op cases I see in practice.

If you're writing hooks

Three things I'd pass on, now that this is behind me:

A hook is a classifier, not a filter. You're building something that answers "is this call a real edit or a no-op?" — the hook infrastructure around it is maybe 10% of the work. The other 90% is the classifier, and the classifier is where every edge case lives.
Write the test for every failure you see in a real session. Not "the kind of failure" — the specific sequence. Keep the Edit JSON, keep the file-before, keep the file-after. Replay them. These replays catch regressions when you change the normalization logic later.
Exit 2 is a feedback channel, not a punishment. Your stderr goes back to Claude as a correction prompt. Think of it as writing an error message for a specific audience that will try to work around it on the next turn. "The edit of foo.py produced no diff" is informative; "if the file already contained what you intended, say so explicitly" is actionable. The second gets the agent to produce a different behavior on the retry.

The full test suite + the hook source is in the public GitHub repo. The packaged install (with the other three hooks and an audit CLI) is Did-It-Actually on Gumroad — $9.50 with the SHOWDEV50 code through Oct 21.

Happy to answer specific "how do you handle X" questions in the comments — the no-op detector has edge cases I probably still haven't seen.

This post was drafted with AI assistance and reviewed for accuracy.

How I stopped Claude Code from lying about finishing tasks (4 hooks, 150 lines of bash)

98dkcc488b-glitch — Wed, 14 Oct 2026 19:19:25 +0000

Last month Claude Code told me it had added a regression test. The commit landed. The PR merged. Two days later a bug slipped through — the test wasn't actually there. The agent had claimed the work, generated a convincing diff summary at the end of the turn, and the tool-call log looked clean. But git log -p showed nothing new in the test file.

That is a specific failure mode worth naming. The model doesn't hallucinate in the "makes up an API" sense. It hallucinates in the "narrates work that didn't happen" sense. Edits silently drop. write_file collapses to a no-op because the content it generated already matched the file. Branch names drift. And the summary at the end of the turn confidently reports what should have happened, not what actually did.

I built a small trust-but-verify layer that catches these at the hook boundary instead of in production. Four hooks, pure bash, zero runtime dependencies. Here's how each one works, and the one non-obvious thing I learned while writing it.

The one Claude Code hook contract fact everything hangs on

Claude Code hooks have an exit-code contract you can use to intervene before and after tool calls. The part worth memorizing is this:

Exit 0 — allow the tool call, swallow stderr.
Exit 2 — block the tool call, and feed stderr back to the model as a correction prompt.

Exit 2 is the whole trick. Your hook doesn't just log a complaint — it tells the model, in-band, why its action was rejected and what to do instead. The model sees your stderr the way it would see any other tool result, and corrects. The agent doesn't need to know a hook exists; it just experiences a world where some actions get a clear "that didn't work, here's why" response.

Once you internalize that, you stop writing "warnings" and start writing hooks that shape the agent's next move.

Note on the snippets. Claude Code hooks receive their input as JSON on stdin (not as $CLAUDE_TOOL_INPUT — thanks to @skillselion in the comments for catching an earlier draft that got this wrong). The snippets below show the real stdin-reading pattern; the full source in the repo uses python3 -c for the trickier parsing but the shape is the same.

Hook 1: verify-write

A PostToolUse hook on Edit and Write. It checks whether the tool call produced any actual change in the working tree.

# simplified; the real hook reads the Claude Code hook payload on stdin.
payload="$(cat)"
file="$(printf '%s' "$payload" | jq -r '.tool_input.file_path // .tool_input.path // empty')"

staged_before="$(cat "$pre_tool_diff_file" 2>/dev/null)"   # written by a sibling PreToolUse
staged_after=$(git diff HEAD -- "$file")

if [ "$staged_before" = "$staged_after" ]; then
  echo "verify-write: the edit of $file produced no change." >&2
  echo "If the file already contained what you intended, say so explicitly instead of re-editing." >&2
  exit 2
fi

Three failure modes it catches in practice:

The no-op edit. The model's old_string and new_string are identical modulo whitespace, so the file writes identical bytes.
The convinced-already edit. The content the model "wanted" to add is already in the file (common after a prior rewrite), so the apply succeeds but changes nothing.
The wrong-file edit. The model edits a backup or a .bak and still reports "updated X".

The non-obvious part: you can't just compare bytes before and after the tool call. Normalization matters. A trailing-newline-only diff is almost always semantic noise and should not count. A whitespace-only diff is a judgment call — I treat it as a real diff by default, because if the model intended a whitespace fix, that's work. But I expose a config flag so a team can tighten it.

Hook 2: block-wrong-branch

A PreToolUse hook on Bash that matches git push and checks whether the current branch is in an allowlist.

payload="$(cat)"
cmd="$(printf '%s' "$payload" | jq -r '.tool_input.command // empty')"

if [[ "$cmd" =~ "git push" ]]; then
  branch=$(git rev-parse --abbrev-ref HEAD)
  allowed=$(jq -r '.allowed_push_branches[]' .claude/config.json 2>/dev/null)
  if ! grep -qx "$branch" <<< "$allowed"; then
    echo "block-wrong-branch: refusing to push $branch." >&2
    echo "This repo only pushes from: $(echo $allowed | tr '\n' ' ')" >&2
    exit 2
  fi
fi

I wrote this after Claude Code pushed what it believed was a feature branch, which was actually main, because it had checked out a different branch halfway through the session and lost track. The hook refuses the push and tells the model what the allowlist is — so the next thing it does is check out the right branch, not re-attempt the same push.

Hook 3: scope-fence

A SessionStart hook that writes a declared-scope file (.claude/.current-scope) and a PreToolUse hook on Edit/Write that rejects any edit to a path outside the fence.

# PreToolUse — hooks receive the full payload on stdin, so pull tool_input from there.
payload="$(cat)"
target=$(printf '%s' "$payload" | jq -r '.tool_input.file_path // empty')
fence=$(cat .claude/.current-scope 2>/dev/null)
if [ -n "$fence" ] && ! [[ "$target" =~ ^($fence)$ ]]; then
  echo "scope-fence: $target is outside the declared scope." >&2
  echo "Current scope: $fence" >&2
  echo "If this edit is intentional, update the scope explicitly." >&2
  exit 2
fi

The fence is a regex set during session start — "today I'm touching src/auth/.* and tests/auth/.*". When Claude Code decides mid-session that it also needs to refactor the ORM layer, the hook catches it. The point isn't that cross-cutting edits are always wrong. The point is that they should be acknowledged. If the agent has to re-declare the scope out loud, I at least get to see it coming instead of discovering it in the diff three turns later.

Hook 4: audit-turn

A Stop hook (runs at the end of a turn, after the final assistant message). It parses the last message for first-person claims — "I added …", "I fixed …", "the tests now pass" — and checks each one against reality:

"I added X" → does git diff show additions in X?
"The tests now pass" → was there a successful test run in the tool-call log for this turn?
"I pushed to branch" → does git log origin/branch..HEAD show the commit?

Mismatches get logged to .claude/trust-audit.log and exit 2 — which, for a Stop hook, refuses the stop and feeds stderr back to Claude, so the agent gets one more turn to actually make the change or correct the claim. (The Stop payload carries stop_hook_active=true the second time around; the hook honours it and lets the stop go through, so an unverifiable claim can't loop forever.) The log also feeds a trust-report CLI that prints a rolling claim-match ratio per repo per week.

I started at 71% in the first week. After two weeks of the first three hooks nudging the agent toward actually doing what it said, the ratio climbed to 94%. Not because the model got better — it's the same model — but because the hooks remove the low-friction path to "lying by narration".

The one surprising lesson

Writing the no-op detector was the hardest part, and I didn't expect that. Deciding what counts as a "real change" has three or four edge cases that each deserve their own test:

whitespace-only edits
trailing-newline-only edits
semantically equivalent reformats (e.g. the model moves an import, nothing else)
write_file that recreates an identical file after a read_file

I ended up shipping 26 integration tests just for the four hooks and the install script, because I couldn't trust the hooks until I'd seen them pass and fail on known inputs. If you build your own, test them. The whole point is to be the thing you trust when you've stopped trusting the model's self-report; your hooks have to earn that trust.

Install

If you want to try it on an existing repo, I've packaged the four hooks plus the trust-report CLI as Did-It-Actually. The install script safely merges into .claude/settings.json with a backup, so you can reverse it. MIT license, works with Claude Code ≥ 1.0.60.

Launch discount for #showdev readers: SHOWDEV50 cuts it to $9.50 — first 25 copies, through Oct 21. The link above applies the code automatically.

If you'd rather just read the no-op detector and crib the idea, the public repo on GitHub has the full source.

Happy to answer questions in the comments — how I handle false positives on whitespace, how the audit-turn parser handles conditional claims ("I think this should pass"), or how to adapt the scope fence for monorepos where "scope" is multiple packages at once.

DEV Community: 98dkcc488b-glitch

How to check if ChatGPT, Gemini, Claude and Perplexity mention your brand (free Python script)

The actual question is simpler than the industry makes it

A script that just does the three

What the three competitors look like at a glance

What the weekly report actually looks like

The failure mode of these tools (including mine)

Code and config

Weekend build: a $19 self-hosted AI visibility tracker (one Python file + GitHub Actions)

What's in the box

What a real run looks like

Why only Gemini and not ChatGPT, Claude, Perplexity

Honest limitations

Launch deal

Day 6: The blind spot in my own blind-spot detector

The two variants

Why this isn't a one-line fix

What I shipped today instead of a fix

Day 5: what 6 strangers found in my 20-line parser

Day 5 of "Did it actually?": what 6 strangers found in my 20-line parser

1. @edtrackai → silent-failure hole, fixed in v1.1

2. @skillselion → two issues, one in the docs and one in the code

3. @launchgatecheck → two detection bugs, one false negative and one false positive

4. @reidmarlow → flag-order bug + git-level flag bypass, both shipped in v1.5

5. @paul_irolla → the one I don't have a shipped fix for

6. @brethofai → tip, not a bug

What this cost and what it bought

A bug in my own branch-guard hook, caught by typing one extra space

A bug in my own branch-guard hook, caught by typing one extra space

The hook

What I typed

Why

The fix

Why I'm writing this up

The hooks, briefly

The kit

26 test cases for a 20-line bash function (and what each one taught me about Claude Code)

Category 1: the "did anything change at all" cases (6 tests)

Category 2: the whitespace cases (5 tests)

Category 3: the Claude-specific cases (6 tests)

Category 4: the git-state cases (5 tests)

Category 5: the integration cases (4 tests)

What this cost me

If you're writing hooks

How I stopped Claude Code from lying about finishing tasks (4 hooks, 150 lines of bash)

The one Claude Code hook contract fact everything hangs on

Hook 1: verify-write

Hook 2: block-wrong-branch

Hook 3: scope-fence

Hook 4: audit-turn

The one surprising lesson

Install

1. `@edtrackai` → silent-failure hole, fixed in v1.1

2. `@skillselion` → two issues, one in the docs and one in the code

3. `@launchgatecheck` → two detection bugs, one false negative and one false positive

4. `@reidmarlow` → flag-order bug + git-level flag bypass, both shipped in v1.5

5. `@paul_irolla` → the one I don't have a shipped fix for

6. `@brethofai` → tip, not a bug