DEV Community: Şahin Uygutalp

How I Detect AI-Generated Text Without Calling an LLM

Şahin Uygutalp — Sat, 28 Mar 2026 11:01:24 +0000

Most AI detection tools make the same mistake: they use an LLM to detect an LLM.

That's expensive, slow, and ironic. You're spending money on the exact technology you're trying to filter out.

For PR-Sentry — a GitHub Action that protects open source maintainers from AI-generated PR spam — I needed something different. Detection had to be free, fast, and impossible to rate-limit. Here's how I built it.

The core insight: AI text has a statistical fingerprint

Human writing is messy. Sentence lengths vary. Word choice is idiosyncratic. Structure is inconsistent.

AI writing is suspiciously uniform. It favors certain words, certain patterns, certain rhythms. Not because it's programmed to — but because it learned from a corpus that rewards this style.

This uniformity is detectable without a model. You just need the right signals.

Signal 1: Buzzword density

AI models consistently overuse a specific vocabulary. Not randomly — these words appear because they score well in RLHF training. "Robust", "seamless", "leverage", "utilize", "comprehensive", "innovative", "streamline".

I built a weighted buzzword list and calculate density per 100 words. A human developer writing a PR description might use one of these. An AI will use four.

def buzzword_density(text):
    words = text.lower().split()
    hits = sum(1 for w in words if w in BUZZWORDS)
    return hits / len(words) * 100

Signal 2: Passive voice ratio

AI consistently overuses passive constructions. "The function is called", "the error is handled", "the issue was fixed". Human developers write more directly.

Passive voice detection is simple with a few regex patterns against auxiliary verb + past participle constructions. Not perfect — but in combination with other signals, it adds real weight.

Signal 3: Sentence length uniformity

This is the subtlest signal and the most reliable. Human writing has high variance in sentence length. Some sentences are short. Others run longer because the thought requires it, building toward a point that the writer is trying to make clearly before moving on.

AI writing has low variance. Everything converges toward a medium length. Calculate the standard deviation of sentence lengths — if it's suspiciously low, something is off.

Signal 4: Repetition score

AI frequently restates the same point in different words. "This fixes the bug. The issue has been resolved. The problem no longer occurs." Count unique trigrams vs total trigrams. High repetition = low uniqueness = suspicious.

Combining the signals

slop_score = (
    buzzword_density * 30 +
    passive_voice_ratio * 20 +
    sentence_length_uniformity * 20 +
    repetition_score * 30
)

is_slop = slop_score >= 60

The weights aren't arbitrary — they reflect how reliably each signal distinguishes AI from human writing in my test corpus. Buzzword density and repetition are the strongest predictors. Sentence uniformity catches cases where the other two don't fire.

Bonus: Shannon entropy for secret detection

This isn't about AI detection — but it's the same "statistics over syntax" philosophy applied to security.

Regex patterns for API keys have a fundamental problem: they only catch known formats. A new service launches, uses a different key format, and your patterns miss it.

Shannon entropy catches anything. Real secrets — API keys, tokens, passwords — are high-entropy strings. Human-readable text is not.

import math
from collections import Counter

def shannon_entropy(s):
    counts = Counter(s)
    length = len(s)
    return -sum(
        (c / length) * math.log2(c / length)
        for c in counts.values()
    )

# Flag anything above 4.5 bits per character
is_suspicious = shannon_entropy(token) > 4.5

AKIA4EXAMPLE123KEY → entropy: 4.7 → flagged.

hello world → entropy: 3.1 → clean.

Does it work?

In testing against a corpus of real PRs and AI-generated PRs, the slop detector catches roughly 70% of AI-generated descriptions with a false positive rate under 5%. It's not perfect — a careful human could write to fool it, and a well-prompted AI can avoid the obvious buzzwords.

But that's fine. The goal isn't perfection. The goal is to avoid calling Claude on every PR that starts with "This PR implements a robust solution to seamlessly address the issue."

The LLM runs on the hard cases. The heuristics handle the obvious ones for free.

The full implementation is in PR-Sentry

If you want to see the complete code — including the full buzzword list, the passive voice patterns, and how this integrates with the GitHub Actions workflow — it's all open source:

→ github.com/Ebuodinde/PR_SENTRY

What signals would you add? I'm especially curious whether anyone has tried perplexity scoring at inference time — the compute cost might be worth it for high-value repos.

I Built a GitHub Action to Stop AI-Generated PRs Before They Reach My Queue

Şahin Uygutalp — Thu, 26 Mar 2026 21:18:13 +0000

Last year, Daniel Stenberg — the author of curl — shut down his project's bug bounty program.

The reason? 20% of the incoming reports were AI-generated garbage. Not just low-quality — worthless. Hallucinated vulnerabilities, copy-pasted exploit templates, fabricated CVEs. His team was spending more time triaging noise than fixing real bugs.

This is the asymmetry nobody talks about: AI can generate 500 lines of plausible-looking code in two seconds. Reviewing it still takes a human hours.

And it's breaking open source.

The industry's fix made things worse

When the "AI PR flood" problem became obvious, the market responded with AI code review bots — CodeRabbit, Copilot review, and friends.

Here's the problem: they review code the way an anxious intern would. They flood your PR timeline with comments about variable naming, whitespace, missing docstrings. They are glorified linters with a chat interface.

Maintainers went from dealing with one source of noise (AI-generated PRs) to dealing with two (AI-generated PRs + AI-generated review comments).

I call this double review fatigue. And it's what made me build something different.

A different approach: Zero-Nitpick

I built PR-Sentry — a GitHub Action with one core rule:

Never comment on style. Only report things that break in production.

That means: security vulnerabilities, runtime crashes, memory leaks, race conditions. Nothing else.

Here's how it works under the hood:

1. Statistical slop detection (no LLM needed)

Before calling any API, PR-Sentry runs a local analysis on the PR description and diff. It calculates a "slop score" based on buzzword density (robust, seamless, leverage, synergy...), passive voice ratio, sentence length patterns, and repetition score.

If the PR scores above 60, it's flagged as AI slop — without burning a single API token.

slop_score = (
    buzzword_density * 30 +
    passive_voice_ratio * 20 +
    sentence_length_avg * 20 +
    repetition_score * 30
)

is_slop = slop_score >= 60

2. Security scanning with entropy analysis

The diff parser checks for 50+ security patterns before any LLM touches the code:

PATTERNS = {
    "aws_key":    r"AKIA[0-9A-Z]{16}",
    "github_pat": r"gh[pousr]_[A-Za-z0-9_]{36,}",
    "openai_key": r"sk-[A-Za-z0-9]{48}",
    "sql_inject": r"SELECT.*FROM.*WHERE.*=.*\$",
    "xss":        r"innerHTML\s*=|document\.write\(",
    # ... 45 more
}

High-entropy strings (Shannon entropy > 4.5) are also flagged to catch accidentally committed secrets.

3. Constrained AI review

Only PRs that pass the slop filter and show signs of potential runtime issues reach the LLM. And the system prompt is strict:

"You are a zero-nitpick code reviewer. Report ONLY: runtime crashes, memory leaks, race conditions, security vulnerabilities. If the code is logically sound, say nothing."

One concise comment. Or silence. Never noise.

Setup takes 2 minutes

Add this to .github/workflows/pr-sentry.yml:

name: PR-Sentry Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run PR-Sentry
        uses: Ebuodinde/PR_SENTRY@v3
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
          # Optional: switch providers
          # provider: openai  # or deepseek

Then add ANTHROPIC_API_KEY to your repo secrets. Done.

No database. No external server. No lock-in — it supports Anthropic, OpenAI, and DeepSeek out of the box.

It also works locally via MCP

If you use Cursor or Claude Code, PR-Sentry ships an MCP server. You can run the slop detector and security scanner against your diff before pushing, directly from your IDE.

What's next

The tool is at v3.0.0 with 262 passing tests. What I want to improve next:

Smarter language-aware slop detection (Python idioms vs JS patterns)
VS Code extension for local pre-push checks
Feedback loop: learning from maintainer decisions over time

If you maintain an open source project and review fatigue is real for you — give it a try. Remove it anytime, it's just a YAML file.

→ github.com/Ebuodinde/PR_SENTRY

Have you noticed an uptick in AI-generated PRs in your repos? Curious how others are handling it — drop a comment.