Verifying Without Judging: Deterministic and Generative Approaches to Code Hallucination Detection

Abstract

Detecting hallucinations in AI-generated code currently admits two broad families of approaches: those that verify against an external source of truth (deterministic) and those that infer the probability of error using models (generative). This article reviews both based on recent literature and experiences published on DEV Community, and situates the specific case of verifying Python packages and APIs against PyPI. The provisional conclusion is that neither family covers the problem completely: deterministic approaches offer objective evidence but limited scope; generative approaches broaden coverage but introduce uncertainty. The choice depends on the type of error one wants to catch and the cost one is willing to assume.

1. The Problem: Three Failures with the Same Name

Saurav Bhattacharya argues in a recent article that hallucination detection is a problem of instrumentation, not of model quality . His central argument is that, in agent systems, the word "hallucination" groups at least three distinct failures:

Parametric leakage: the agent responds from its training memory instead of using the result of the tool it was given. The answer may be correct, but it is correct by chance, not because it is grounded.

Fabricated grounding: the agent cites a source, an identifier, or a number that does not appear in the retrieved context. This is the most dangerous case because it has the appearance of a documented claim.

Unsupported synthesis: each individual fact is in the context, but the conclusion the agent draws does not follow from the source.

The practical implication is that these three failures require different detectors. A single "hallucination score" that mixes them produces a figure that no one can interpret .

1. The Critique of Generative Methods

The same article points out a known weakness of model-based detectors: self-consistency (sampling several times and checking whether the answers match) detects unstable hallucinations but fails with stable ones. If an agent systematically reproduces the same erroneous datum from its parametric memory, the samples match and the detector reports high confidence. The model is reproducibly incorrect, and consistency was the signal .

The academic literature supports this caution. A study on static analysis methods for library hallucinations notes that type-inference tools such as Pyright are more reliable than LLM judgment for detecting import and type errors . The same work acknowledges, however, that static analysis has limits: data-flow and control-flow operations are the most promising avenues for broadening coverage, but types can mutate during execution and escape detection .

1. The Deterministic Proposal: Verifying Against the Source

The alternative Bhattacharya advocates is mechanical and explainable: extract the verifiable claims from the output and check them against the text the agent actually retrieved. The requirement is to preserve that text (the execution trace); without it, any verification is a guess .

In the code domain, this idea has materialized in several forms. The libhallubench tool evaluates LLM code generation against a benchmark of nonexistent libraries, using the PyPI package list as the source of truth . The vibe-check project detects typical patterns of AI-generated code, including invented imports, through pattern matching, without resorting to models or cloud services . An article from the same community describes how ESLint and TypeScript fail to detect an invented import because the linter validates syntax, not the existence of the package in the registry, and the type compiler only checks types if installed definitions exist, which cannot be installed for a package that does not exist .

Within this landscape, verifying Python packages and APIs against the actual PyPI wheel sits in the deterministic tradition: it does not infer, it checks set membership. The advantage is that it requires neither installing the package nor executing the analyzed code; the limitation is that it only covers the existence of symbols, not the correctness of their use.

1. The Generative Proposal: Detecting Spans with Trained Models

The generative family has advanced notably. LettuceDetect v2 is a detector trained on a unified benchmark of 74,285 examples labeled at the span level, covering code, tool outputs, and documents . The generative model (Qwen 3.5-2B) returns hallucinated spans as structured JSON, typed by category (contradiction, unsupported addition, fabricated reference) and subcategory (entity, temporal, numeric, identifier, attribute) . On the unified test set, it reaches 0.689 span-F1; on code-agent responses, 0.602 .

The advantage of this approach is coverage: it detects errors that an existence verifier cannot catch, such as an incorrect logical condition or a wrong numeric value. The limitation is inherent to any probabilistic model: it does not offer a deterministic explanation and its performance varies by domain and grounding source .

1. The Gap Neither Covers: Logic and Resources

The CodeHalu taxonomy, cited in the documentation of the project discussed here, distinguishes four categories: naming, mapping, resources, and logic. Deterministic verifiers against registries reasonably cover the first two. The last two fall outside their scope by definition: they require analyzing data flow, conditions, or algorithmic complexity, not just the existence of a symbol.

The literature confirms this limit. An analysis of structural failures in generated code concludes that most of them escape type checking, tests, and static security analysis . Another work describes the case of a discount function whose types are correct but whose semantics are wrong because the percentage unit does not match the one used in the rest of the code: the compiler cannot detect it because the types line up .

Generative detectors such as LettuceDetect v2 attempt to cover part of that gap, but they do not offer deterministic guarantees. The gap between "the symbol exists" and "the code does what it should" remains, to a large extent, an open problem.

1. Considerations for Detector Design

From the above, some provisional observations can be drawn:

Deterministic verification is cheap and explainable. Checking the existence of a package or function against a registry requires neither executing the user's code nor installing dependencies. The cost is bounded and the result is a set membership, not a probability.

Deterministic coverage is necessarily partial. It is limited to errors that can be expressed as a membership or signature check. Logic, data-flow, and complexity errors fall outside.

Generative methods broaden coverage but introduce uncertainty. Their performance depends on the training benchmark and they do not offer a deterministic explanation of the failure.

Combining them is plausible. A deterministic detector for symbol existence and a generative or semantic detector for logic could cover complementary layers. The open question is how to integrate them without the uncertainty of the second contaminating the reliability of the first.

1. Limitations of This Analysis

This text is based on a non-systematic review of literature and community articles. No comparative empirical evaluation of the described approaches has been carried out. The figures cited come from the original works and have not been replicated in the specific context of Python package verification. The CodeHalu taxonomy has been used as a descriptive framework, not as a validated benchmark in this analysis.
References

Bhattacharya, S. (2026). Hallucination Is Not a Vibe: How to Actually Detect Ungrounded Claims in Agent Output. DEV Community.

libhallubench (2026). PyPI.

5 Code Smells Only AI Creates (And How to Detect Them). DEV Community.

ESLint can't catch this: 5 failure modes unique to AI-generated code. DEV Community.

LettuceDetect v2 in Semantic Router: Generative Hallucination Detection as a vLLM Endpoint. GitHub.

KRLabsOrg. LettuceDetect. GitHub.

The verification gap in agent-written code: what actually breaks. GitHub.

The Patchwork Problem in LLM-Generated Code. arXiv.

For mitigation, we use three metrics. OpenReview.