I Built a Lie Detector for LLM Output — It Found 4 Functions That Don't Exist in My Codebase

I was deep in a refactor last month, and I did what any reasonable developer does when the code gets messy: I asked an LLM for help.

I pasted in a few hundred lines of Python, explained the context, and got back a beautifully formatted response. The model was confident. It referenced get_user_profile(), suggested I move logic into DataProcessor, and even told me to check src/helpers/formatter.py for the existing abstraction.

So I went looking.

None of them existed. Not a single one. I spent twenty minutes grepping, clicking through files, wondering if I had a branch mismatch, before it finally hit me: the model had simply made them up. It wrote about my codebase with the calm authority of someone who had absolutely no idea what was in it.

That was the third time in a week. I decided to build a fix.

The Problem Nobody Talks About Enough

LLMs hallucinate code references constantly. Everyone knows this in the abstract, but the actual experience of it is more annoying than the discourse suggests — because the hallucinated references are plausible. They look like real function names. They follow your naming conventions. They almost make sense.

And there's no automated way to catch them. You either notice because you know your codebase cold, or you waste time chasing ghosts.

What I wanted was something I could pipe LLM output through and get a straight answer: does this stuff actually exist?

Meet hallucination-grep

pip install hallucination-grep

# pipe LLM output directly (macOS clipboard example)
pbpaste | hallucination-grep --codebase ./src

# or point it at a saved response file
hallucination-grep response.txt --codebase .

# output JSON for CI pipelines
hallucination-grep response.txt --codebase . --format json

That's the entire interface. Give it LLM text, point it at your codebase, get a report.

What the Output Looks Like

hallucination-grep — scanning response.txt against ./src

  Checking 312 lines of LLM output...
  Indexed 47 Python files | 203 functions | 31 classes

  HALLUCINATIONS FOUND (4)
  ─────────────────────────────────────────────────────

  [FUNCTION]  get_user_profile()
    Referenced at line 14 of LLM output
    Not found in codebase
    Closest match: get_user_data() in src/users/service.py

  [CLASS]     DataProcessor
    Referenced at line 27 of LLM output
    Not found in codebase
    Closest match: DataHandler in src/pipeline/core.py

  [FILE]      src/helpers/formatter.py
    Referenced at line 41 of LLM output
    File does not exist
    Closest match: src/utils/formatting.py

  [IMPORT]    from utils.transform import normalize_records
    Referenced at line 58 of LLM output
    normalize_records not found in utils/transform
    Closest match: normalize_data in src/utils/transform.py

  ─────────────────────────────────────────────────────
  Result: 4 hallucinations detected. Exit code: 1

The "closest match" suggestions are the part that makes this actually useful rather than just accusatory. It's not enough to know the model was wrong — you want to know what it probably meant.

How It Works

The tool has four stages, all pure Python, no external APIs.

1. Reference extraction — Regex patterns scan the LLM text for code references: function calls (word()), class names (PascalCase tokens), file paths (src/something/file.py), import statements, and method chains. This is intentionally aggressive. False positives get filtered in the next step.

2. Codebase indexing — Python's ast module walks every .py file in your target directory. It catalogs every function definition, class definition, method, module-level variable, and the file paths themselves. This gives you a ground-truth index of what actually exists.

3. Cross-referencing — Every extracted reference gets checked against the index. If it's not there, it's flagged. The check is case-sensitive for imports and class names, case-insensitive for function names (since LLMs sometimes drift on capitalization).

4. Similarity matching — For each hallucination, difflib.get_close_matches() finds the nearest real identifier by edit distance. This is what produces the "Closest match" suggestions. It's not fancy, but it works surprisingly well for catching the typical LLM pattern of near-misses.

The whole thing runs locally. No network calls. A medium-sized codebase (a few hundred files) indexes in under a second.

The CI Integration Angle

The exit code behavior is deliberate. When hallucinations are found, the tool exits with code 1. When the LLM output is clean, it exits 0.

That means you can drop it into a GitHub Actions workflow:

- name: Validate LLM output
  run: |
    hallucination-grep llm_response.txt --codebase ./src --format json

If you're using LLMs to generate code review comments, documentation suggestions, or refactor proposals in a pipeline, you can gate on this check before the output gets acted on. The JSON output mode makes it easy to parse results downstream.

What I Learned Building This

• AST analysis is underused. The ast module is right there in the stdlib, and it gives you a complete structural view of Python code. Most people reach for regex when they need to analyze code; AST is almost always the better choice.

• Regex for extraction, AST for indexing — these two tools have different strengths. Regex is good at finding patterns in prose (LLM output). AST is good at understanding code structure. Using each for what it's suited for made the architecture clean.

• difflib is genuinely good. I expected to need something fancier for the similarity matching. The SequenceMatcher approach in difflib handles the typical hallucination pattern (slightly wrong function name, transposed words) better than I expected.

• LLMs are more wrong about imports than anything else. In my testing, import hallucinations were the most common failure mode. The model will confidently suggest from utils.helpers import process_data when neither the module path nor the function exists. These are also the most dangerous because they fail silently at runtime until you actually hit that code path.

• Piped input is underappreciated as a UX pattern. Designing the tool to work with pbpaste | or from clipboard managers made it feel native to the terminal workflow in a way that file-only tools don't.

Try It

The tool is MIT licensed and the code is straightforward enough to read in an afternoon.

pip install hallucination-grep

GitHub: https://github.com/LakshmiSravyaVedantham/hallucination-grep

If you've been burned by hallucinated code references — and if you use LLMs for development work, you have been — give it a run on the next response you get. The first time it catches something the model said with complete confidence, it's a little vindicating.

Pull requests and issues welcome. Especially interested in support for other languages beyond Python.