Back in June I wrote about a real bug in this repo: claude -p-generated commit messages kept slipping Co-Authored-By: Claude lines past an explicit "never do this" instruction in both the system prompt and CLAUDE.md. Telling the model not to sign its own work wasn't enough — it did it anyway, often enough to matter. The fix was a deterministic filter applied after generation, not another instruction hoping the model listens:
_STRIP = ("co-author", "co-authored", "generated by", "claude", "anthropic", "openai", "llm", "ai:")
def _claude(prompt: str, system: str = None) -> str:
full = (system + "\n\n" + prompt) if system else prompt
raw = subprocess.check_output(["claude", "-p", full], text=True).strip()
return "\n".join(
l for l in raw.splitlines()
if not any(s in l.lower() for s in _STRIP)
).strip()
Any line containing any of those substrings, case-insensitive, gets dropped entirely. It's used by both git_commit.py (the standalone CLI) and server.py's generate_commit_message MCP tool — same helper, same filter, both paths covered. I called that fixed and moved on, and it's held up: I haven't seen a signed commit slip through since.
What I never did was ask the opposite question: not "does it catch everything it should," but "does it catch things it shouldn't." A plain substring filter doesn't know about word boundaries or context. It just asks "is this string anywhere in this line," and "llm" is a three-character substring that shows up in perfectly ordinary technical writing — including, unfortunately, writing about this exact codebase, which has an MCP tool called generate_commit_message that literally calls an LLM.
I tested it against commit subjects this repo's own commit-message generator could plausibly produce, given what's actually in server.py and git_commit.py:
_STRIP = ("co-author", "co-authored", "generated by", "claude", "anthropic", "openai", "llm", "ai:")
def apply_strip(raw):
return "\n".join(
l for l in raw.splitlines()
if not any(s in l.lower() for s in _STRIP)
).strip()
apply_strip("fix: retry llm calls on 429 with backoff")
# -> ''
apply_strip("feat: add llm-based tag scoring for trending posts")
# -> ''
apply_strip("docs: explain how generate_commit_message calls the llm subprocess")
# -> ''
apply_strip("chore: bump mcp[cli] and pin llm response timeout")
# -> ''
Every single one comes back empty. Not trimmed, not partially redacted — the whole line is gone, because the whole line was the only line, and the whole line contained "llm". If _claude() had genuinely generated any of those as its one-line commit subject, the function's return value would be an empty string. Not a warning, not a fallback subject, not an exception — just "", silently, and whatever calls _claude() gets to decide what an empty commit message means with zero signal that a filter did it on purpose.
The original bug was under-filtering: an instruction that didn't reliably stop AI attribution from appearing. This is the mirror image, over-filtering: a mechanical fix precise enough to stop attribution but blunt enough to also erase legitimate technical content that happens to share a substring with the thing it's hunting. Both bugs live in the same eight-item tuple, and neither is really about AI at all — "ai:" as a strip target would just as happily eat a line like "detail: rewrite retry logic" if a future edit ever adds a colon-suffixed word ending in those two letters. Substring matching over free text is the mechanism, and it doesn't distinguish "the model is claiming credit" from "the model is describing its own subject matter accurately."
The honest reason I never caught this earlier: git_commit.py and generate_commit_message have both been running against my own real commits for a month, and none of my commits so far have needed the word "llm" in the subject line — I tend to write "ai commit script" or "claude cli" when the diff touches that code, and those get caught by different, more deliberate entries in the tuple ("claude") doing the job they were meant to do. The gap was invisible because the specific overlap — a commit whose subject matter is the LLM feature itself — never actually came up. That's the same shape as the detached-HEAD guard clause I wrote up a few days ago: a branch that's correct in principle but has never been exercised by a real input isn't verified, it's just unfalsified.
The actual fix is narrower matching, not a different mechanism. "llm" and "ai:" are the two entries doing collateral damage — they're topic words, not attribution phrases, and they're in the tuple because early testing saw the model occasionally write things like "as an AI, I should note" or "using LLM-generated content" as a hedge before the actual message. The other six entries ("co-author", "co-authored", "generated by", "claude", "anthropic", "openai") are all either full phrases or proper nouns unlikely to appear in a legitimate commit about unrelated work in this repo. Swapping the two topic words for the actual attribution phrases they were standing in for — "as an ai", "llm-generated" — keeps the filter's job (strip lines where the model is talking about itself) without also stripping every line where the model is correctly describing what the commit touches. I haven't shipped that change yet; today's finding is the audit, not the patch, and I'd rather land the narrower patterns against a few more real generated messages first than guess at the replacement strings from a handful of hand-written test cases.
If your project has a deterministic filter standing in for a natural-language instruction — and after the last one, this repo has several — the question worth asking isn't just "did this catch the bug I built it for." It's "what does this filter's own vocabulary look like as a search pattern over content my project actually produces," because a tuple of bare substrings doesn't know the difference between a word used as a name and the same word used as a topic.
Top comments (0)