DEV Community

Enjoy Kumawat
Enjoy Kumawat

Posted on

My Commit Script Was Fixed to Never Say "As an AI." My MCP Tool's Copy of the Same Function Wasn't.

I run an MCP server (server.py) that exposes a generate_commit_message tool, and a standalone CLI (git_commit.py) that does the same job from a prepare-commit-msg git hook. Back on 2026-06-21 I wrote an article claiming these two copies "run the exact same code" — same claude -p subprocess call, same regex filter stripping AI attribution lines out of whatever the model returns. Five weeks and two more regex-hardening passes later, I finally checked the one thing none of those passes ever looked at: the actual instructions each copy sends to the model before it generates anything. They have never matched, since the day I supposedly fixed both.

Here's how I found it, and why "same filter" turned out to be doing a lot of hiding.

The filter everyone kept syncing

Both files have a _STRIP_RE (originally _STRIP, a bare-substring tuple) that runs over the model's raw output and drops any line that looks like AI attribution:

_STRIP_PATTERNS = [
    r"co-authored-by\s*:",
    r"generated (with|by)\s+claude",
    r"\bclaude code\b",
    r"\bwritten by (an )?(ai|llm|claude|chatgpt|copilot)\b",
    r"\bai-generated\b",
    r"🤖",
]
_STRIP_RE = re.compile("|".join(_STRIP_PATTERNS), re.IGNORECASE)
Enter fullscreen mode Exit fullscreen mode

This tuple has been touched three times since it was introduced: once on 2026-06-21 (created), once on 2026-07-22 (found it over-matched real commits about the project's own LLM code — "retry llm calls on 429" was getting nuked), once on 2026-07-26 (rewrote it as word-boundary regex). Every single time, both copies got edited together, in the same commit, kept byte-identical. I checked today — they still are, character for character.

Because that filter got so much attention, I'd assumed it was the whole safety story. It isn't. It's the mop, not the instruction not to spill.

What the model is actually told

git_commit.py's system prompt:

SYSTEM = (
    "You are a git commit message generator. "
    "Output ONLY the commit message — one line, no explanation, no markdown, no quotes, "
    "no co-author lines, no signatures, no AI references. "
    "Follow Conventional Commits: type(scope): subject. "
    "Types: feat, fix, docs, style, refactor, test, chore. "
    "Subject: imperative, lowercase, max 72 chars."
)
Enter fullscreen mode Exit fullscreen mode

server.py's generate_commit_message tool, before today:

return _claude(
    diff,
    system=(
        "You are a git commit message generator. "
        "Output ONLY the commit message — no explanation, no markdown, no quotes. "
        "Follow Conventional Commits: type(scope): subject. "
        "Types: feat, fix, docs, style, refactor, test, chore. "
        "Subject: imperative, lowercase, max 72 chars."
    ),
)
Enter fullscreen mode Exit fullscreen mode

Diff the two: git_commit.py tells the model up front — one line, no co-author lines, no signatures, no AI references. server.py never did. It relies entirely on the regex catching whatever slips out downstream.

How they forked

I went back through git history instead of trusting memory. Both functions were added in the same commit, 2026-06-21, feat: AI commit message generator using Claude CLI. The very next commit, same day, fix: strip AI attribution lines from generated commit messages (1f8808c), touched both files — but not the same way:

diff --git a/git_commit.py b/git_commit.py
-    "Output ONLY the commit message — no explanation, no markdown, no quotes. "
+    "Output ONLY the commit message — one line, no explanation, no markdown, no quotes, "
+    "no co-author lines, no signatures, no AI references. "
Enter fullscreen mode Exit fullscreen mode
diff --git a/server.py b/server.py
+_STRIP = ("co-author", "co-authored", "generated by", "claude", "anthropic", "openai", "llm", "ai:")
 def _claude(prompt: str, system: str = None) -> str:
     full = (system + "\n\n" + prompt) if system else prompt
-    return subprocess.check_output(["claude", "-p", full], text=True).strip()
+    raw = subprocess.check_output(["claude", "-p", full], text=True).strip()
+    return "\n".join(...)
Enter fullscreen mode Exit fullscreen mode

Same commit, same message, same day. One file got a stronger system prompt. The other got the regex added to its shared helper function, but its own generate_commit_message tool — the caller of that helper — kept the old, unstrengthened instructions. I don't think this was a deliberate choice; I think _claude() and SYSTEM looked like the same kind of edit from inside one commit, and only one of the two actually was.

Why the regex alone isn't enough

I tested what the current shared regex actually catches, against phrasing a model is more likely to produce when it hasn't been told not to:

tests = [
    "fix: retry llm calls on 429 with backoff",
    "As an AI language model, I cannot take credit for this commit.",
    "I am an AI and generated this change automatically.",
    "fix: handle claude api rate limit in retry loop",
]
for t in tests:
    print(bool(STRIP_RE.search(t)), "|", t)
Enter fullscreen mode Exit fullscreen mode
False | fix: retry llm calls on 429 with backoff
False | As an AI language model, I cannot take credit for this commit.
False | I am an AI and generated this change automatically.
False | fix: handle claude api rate limit in retry loop
Enter fullscreen mode Exit fullscreen mode

The first and fourth lines correctly survive — they're legitimate commits about this project's own LLM-calling code, which is exactly what the 2026-07-22/07-26 fixes were trying to preserve. But the second and third lines are the failure mode the instruction was meant to prevent at the source, and they walk straight past the filter downstream, because none of the six patterns match generic first-person AI disclosure — only specific attribution phrasings like "generated by claude" or "co-authored-by:". git_commit.py's prompt makes this class of output much less likely to be generated in the first place. server.py's tool, until today, had no such defense — just the same mop, waiting for a spill the mop was never built to catch.

The fix

One line, matching what git_commit.py already had:

system=(
    "You are a git commit message generator. "
    "Output ONLY the commit message — one line, no explanation, no markdown, no quotes, "
    "no co-author lines, no signatures, no AI references. "
    "Follow Conventional Commits: type(scope): subject. "
    "Types: feat, fix, docs, style, refactor, test, chore. "
    "Subject: imperative, lowercase, max 72 chars."
),
Enter fullscreen mode Exit fullscreen mode

Nothing clever. The interesting part isn't the fix, it's how long the gap survived scrutiny that should have caught it. This exact codepath has had a dedicated article about its duplication (whether it's "the same code"), two dedicated articles hardening its regex filter, and multiple comment replies confirming the STRIP tuple stayed in sync across both files. Every one of those checks looked at the same artifact — the filter — because that's the thing that had a name and a shared constant to diff. The system prompt was just an inline string argument, invisible unless you go looking for it specifically, and across five weeks and four separate touchpoints on this exact function pair, nobody did.

If you maintain two copies of "the same" AI-calling code, the thing worth diffing isn't just the post-processing step you already keep in sync — it's the prompt itself, the part with no shared constant forcing you to notice when it drifts.

Top comments (0)