<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Carlos Luque Zayas Bazan</title>
    <description>The latest articles on DEV Community by Carlos Luque Zayas Bazan (@carlos_luquezayasbazan_).</description>
    <link>https://dev.to/carlos_luquezayasbazan_</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4209543%2Fcf4d8e98-ad29-402c-8a76-a6ed29cd3e49.png</url>
      <title>DEV Community: Carlos Luque Zayas Bazan</title>
      <link>https://dev.to/carlos_luquezayasbazan_</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/carlos_luquezayasbazan_"/>
    <language>en</language>
    <item>
      <title>Verifying Without Judging: Deterministic and Generative Approaches to Code Hallucination Detection</title>
      <dc:creator>Carlos Luque Zayas Bazan</dc:creator>
      <pubDate>Fri, 06 Nov 2026 16:53:17 +0000</pubDate>
      <link>https://dev.to/carlos_luquezayasbazan_/verifying-without-judging-deterministic-and-generative-approaches-to-code-hallucination-detection-4p2d</link>
      <guid>https://dev.to/carlos_luquezayasbazan_/verifying-without-judging-deterministic-and-generative-approaches-to-code-hallucination-detection-4p2d</guid>
      <description>&lt;p&gt;Abstract&lt;/p&gt;

&lt;p&gt;Detecting hallucinations in AI-generated code currently admits two broad families of approaches: those that verify against an external source of truth (deterministic) and those that infer the probability of error using models (generative). This article reviews both based on recent literature and experiences published on DEV Community, and situates the specific case of verifying Python packages and APIs against PyPI. The provisional conclusion is that neither family covers the problem completely: deterministic approaches offer objective evidence but limited scope; generative approaches broaden coverage but introduce uncertainty. The choice depends on the type of error one wants to catch and the cost one is willing to assume.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The Problem: Three Failures with the Same Name&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Saurav Bhattacharya argues in a recent article that hallucination detection is a problem of instrumentation, not of model quality . His central argument is that, in agent systems, the word "hallucination" groups at least three distinct failures:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Parametric leakage: the agent responds from its training memory instead of using the result of the tool it was given. The answer may be correct, but it is correct by chance, not because it is grounded.

Fabricated grounding: the agent cites a source, an identifier, or a number that does not appear in the retrieved context. This is the most dangerous case because it has the appearance of a documented claim.

Unsupported synthesis: each individual fact is in the context, but the conclusion the agent draws does not follow from the source.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;The practical implication is that these three failures require different detectors. A single "hallucination score" that mixes them produces a figure that no one can interpret .&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The Critique of Generative Methods&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The same article points out a known weakness of model-based detectors: self-consistency (sampling several times and checking whether the answers match) detects unstable hallucinations but fails with stable ones. If an agent systematically reproduces the same erroneous datum from its parametric memory, the samples match and the detector reports high confidence. The model is reproducibly incorrect, and consistency was the signal .&lt;/p&gt;

&lt;p&gt;The academic literature supports this caution. A study on static analysis methods for library hallucinations notes that type-inference tools such as Pyright are more reliable than LLM judgment for detecting import and type errors . The same work acknowledges, however, that static analysis has limits: data-flow and control-flow operations are the most promising avenues for broadening coverage, but types can mutate during execution and escape detection .&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The Deterministic Proposal: Verifying Against the Source&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The alternative Bhattacharya advocates is mechanical and explainable: extract the verifiable claims from the output and check them against the text the agent actually retrieved. The requirement is to preserve that text (the execution trace); without it, any verification is a guess .&lt;/p&gt;

&lt;p&gt;In the code domain, this idea has materialized in several forms. The libhallubench tool evaluates LLM code generation against a benchmark of nonexistent libraries, using the PyPI package list as the source of truth . The vibe-check project detects typical patterns of AI-generated code, including invented imports, through pattern matching, without resorting to models or cloud services . An article from the same community describes how ESLint and TypeScript fail to detect an invented import because the linter validates syntax, not the existence of the package in the registry, and the type compiler only checks types if installed definitions exist, which cannot be installed for a package that does not exist .&lt;/p&gt;

&lt;p&gt;Within this landscape, verifying Python packages and APIs against the actual PyPI wheel sits in the deterministic tradition: it does not infer, it checks set membership. The advantage is that it requires neither installing the package nor executing the analyzed code; the limitation is that it only covers the existence of symbols, not the correctness of their use.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The Generative Proposal: Detecting Spans with Trained Models&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The generative family has advanced notably. LettuceDetect v2 is a detector trained on a unified benchmark of 74,285 examples labeled at the span level, covering code, tool outputs, and documents . The generative model (Qwen 3.5-2B) returns hallucinated spans as structured JSON, typed by category (contradiction, unsupported addition, fabricated reference) and subcategory (entity, temporal, numeric, identifier, attribute) . On the unified test set, it reaches 0.689 span-F1; on code-agent responses, 0.602 .&lt;/p&gt;

&lt;p&gt;The advantage of this approach is coverage: it detects errors that an existence verifier cannot catch, such as an incorrect logical condition or a wrong numeric value. The limitation is inherent to any probabilistic model: it does not offer a deterministic explanation and its performance varies by domain and grounding source .&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The Gap Neither Covers: Logic and Resources&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The CodeHalu taxonomy, cited in the documentation of the project discussed here, distinguishes four categories: naming, mapping, resources, and logic. Deterministic verifiers against registries reasonably cover the first two. The last two fall outside their scope by definition: they require analyzing data flow, conditions, or algorithmic complexity, not just the existence of a symbol.&lt;/p&gt;

&lt;p&gt;The literature confirms this limit. An analysis of structural failures in generated code concludes that most of them escape type checking, tests, and static security analysis . Another work describes the case of a discount function whose types are correct but whose semantics are wrong because the percentage unit does not match the one used in the rest of the code: the compiler cannot detect it because the types line up .&lt;/p&gt;

&lt;p&gt;Generative detectors such as LettuceDetect v2 attempt to cover part of that gap, but they do not offer deterministic guarantees. The gap between "the symbol exists" and "the code does what it should" remains, to a large extent, an open problem.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Considerations for Detector Design&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;From the above, some provisional observations can be drawn:&lt;/p&gt;

&lt;p&gt;Deterministic verification is cheap and explainable. Checking the existence of a package or function against a registry requires neither executing the user's code nor installing dependencies. The cost is bounded and the result is a set membership, not a probability.&lt;/p&gt;

&lt;p&gt;Deterministic coverage is necessarily partial. It is limited to errors that can be expressed as a membership or signature check. Logic, data-flow, and complexity errors fall outside.&lt;/p&gt;

&lt;p&gt;Generative methods broaden coverage but introduce uncertainty. Their performance depends on the training benchmark and they do not offer a deterministic explanation of the failure.&lt;/p&gt;

&lt;p&gt;Combining them is plausible. A deterministic detector for symbol existence and a generative or semantic detector for logic could cover complementary layers. The open question is how to integrate them without the uncertainty of the second contaminating the reliability of the first.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Limitations of This Analysis&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This text is based on a non-systematic review of literature and community articles. No comparative empirical evaluation of the described approaches has been carried out. The figures cited come from the original works and have not been replicated in the specific context of Python package verification. The CodeHalu taxonomy has been used as a descriptive framework, not as a validated benchmark in this analysis.&lt;br&gt;
References&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Bhattacharya, S. (2026). Hallucination Is Not a Vibe: How to Actually Detect Ungrounded Claims in Agent Output. DEV Community.

libhallubench (2026). PyPI.

5 Code Smells Only AI Creates (And How to Detect Them). DEV Community.

ESLint can't catch this: 5 failure modes unique to AI-generated code. DEV Community.

LettuceDetect v2 in Semantic Router: Generative Hallucination Detection as a vLLM Endpoint. GitHub.

KRLabsOrg. LettuceDetect. GitHub.

The verification gap in agent-written code: what actually breaks. GitHub.

The Patchwork Problem in LLM-Generated Code. arXiv.

For mitigation, we use three metrics. OpenReview.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>programming</category>
      <category>python</category>
    </item>
    <item>
      <title>The Registry That Wasn't Enough: How We Went from "Knowing APIs" to Actually Verifying Against PyPI</title>
      <dc:creator>Carlos Luque Zayas Bazan</dc:creator>
      <pubDate>Tue, 03 Nov 2026 19:29:46 +0000</pubDate>
      <link>https://dev.to/carlos_luquezayasbazan_/the-registry-that-wasnt-enough-how-we-went-from-knowing-apis-to-actually-verifying-against-pypi-3j8</link>
      <guid>https://dev.to/carlos_luquezayasbazan_/the-registry-that-wasnt-enough-how-we-went-from-knowing-apis-to-actually-verifying-against-pypi-3j8</guid>
      <description>&lt;p&gt;The Registry That Wasn't Enough: How We Went from "Knowing APIs" to Actually Verifying Against PyPI&lt;/p&gt;

&lt;p&gt;Continuation of the previous article on detecting hallucinations in AI-generated code.&lt;/p&gt;

&lt;p&gt;In the previous article, I told the story of how this idea was born: a requests.get_json() that never existed, that passed tests, linter, and type checker, and that would only have failed in production. The tool I built back then relied on a handcrafted registry of valid APIs: a map of library → known functions, and a simple rule: if the library is in the registry and the function isn't, it's a hallucination. It worked for requests, json, and os, and it detected nonexistent packages by querying PyPI.&lt;/p&gt;

&lt;p&gt;But the registry had an obvious limitation: it only judged what it knew. And as soon as you stepped outside requests, json, and os, the tool went silent. Not because there were no hallucinations, but because it had nothing to compare against. That ceiling is what I've been thinking about for the past few weeks. And this is what came next.&lt;/p&gt;

&lt;p&gt;Of course, this work doesn't start from scratch. Recent research confirms that library hallucinations are systematic (5.2% of packages mentioned by commercial models don't exist, and up to 21.7% for open-source models), that static analysis has a ceiling of 48.5% to 77% of what it can possibly detect, and that verification against real documentation significantly reduces false positives. Our project addresses some of those limitations (invented packages, APIs of real packages verified against wheels) and leaves others documented (APIs of nonexistent packages, advanced slopsquatting). This continuation covers what we've done and what remains.&lt;br&gt;
How to Verify More Rigorously?&lt;/p&gt;

&lt;p&gt;The question that changed everything was: what if, instead of maintaining a hand-written list of valid functions, we read what the package actually contains?&lt;/p&gt;

&lt;p&gt;PyPI doesn't just serve packages: it serves wheels, which are ZIP files containing the code and — very importantly — the type stubs (.pyi) when the package includes them. If you download the wheel and open it, you can check whether class DataFrame exists in pandas without installing anything and without executing any of the user's code.&lt;/p&gt;

&lt;p&gt;That's exactly what we did. And it changes the game: you no longer depend on someone maintaining a list of functions by hand. You depend on what the package actually publishes on PyPI. Coverage stops being artisanal and becomes that of the ecosystem.&lt;br&gt;
The Cache, or How Not to Punish PyPI on Every Analysis&lt;/p&gt;

&lt;p&gt;The first practical problem was obvious: if every analysis queries PyPI for every import, on a slow network like mine, the analysis drags on forever. And if PyPI goes down, the detector just sits there waiting. So we built a two-level cache (memory + disk) with TTLs:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;exists (does the package exist on PyPI?): 7 days.

Reputation metadata (first release, maintainers): 30 days.

Network errors: 10 minutes (to avoid retrying in a loop when the network is flaky).
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;And a design decision that was already there in the previous article but now becomes central: fail-open. If we can't verify (timeout, PyPI down), we don't report the package as nonexistent. Better an occasional false negative than a false positive that erodes trust.&lt;/p&gt;

&lt;p&gt;The result is noticeable: the second run on the same file goes from ~50 seconds to 17 milliseconds. And all of that without touching the normal flow: if you don't configure the cache, the detector works exactly as before.&lt;br&gt;
Canonicalization, stdlib, and the False Positive That Taught Us Something&lt;/p&gt;

&lt;p&gt;With the cache came the fine details. PyPI normalizes names (PIL is installed as Pillow, sklearn as scikit-learn, cv2 as opencv-python), and imports bring submodules (tensorflow.keras, os.path). If you don't canonicalize, the cache misses and odd false positives start appearing.&lt;/p&gt;

&lt;p&gt;One of the most instructive: imp (a stdlib module, deprecated) was flagged as typosquatting of pip. The edit distance was 1, and the heuristic didn't know that imp is stdlib. The lesson: stdlib lists and PyPI package lists must stay in sync, or the detector starts inventing problems where there are none. Today imp, distutils, and friends are recognized as stdlib and are not flagged.&lt;br&gt;
Actually Verifying Attributes: pandas.DataFrame Is No Longer a False Positive&lt;/p&gt;

&lt;p&gt;Here's the biggest change compared to the previous article. Before, api-hallucination was heuristic: it looked for suspicious names, it didn't verify. And that produced false positives like pandas.DataFrame, which does exist but wasn't in the registry.&lt;/p&gt;

&lt;p&gt;Now there's a deep mode (deep) that, when you enable it:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Queries PyPI for the package.

Downloads the wheel (once, cached per version).

Opens the ZIP and searches for the attribute in all the package's .pyi and .py files.

Returns one of three states: confirmed, not found, or unverifiable.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;The result, measured with the same test file: 15 findings without deep mode, 14 with deep mode. The difference is exactly the false positive for pandas.DataFrame, which disappears. And more importantly: pandas.funcion_inventada is detected, with evidence that it's not in the real wheel.&lt;/p&gt;

&lt;p&gt;That's something the handcrafted registry could never provide. And we did it without installing Python on the server, without executing the user's code, and without adding third-party dependencies: just HTTP, ZIP, and a bit of regex.&lt;br&gt;
IP Control, or How to Open It Up Without Getting Your Server Knocked Over&lt;/p&gt;

&lt;p&gt;Deep mode downloads wheels. That costs bandwidth and time. Before opening it to the public, we needed a limit. So we added IP-based control: 3 deep analyses per day per IP, with a daily reset.&lt;/p&gt;

&lt;p&gt;Three details that mattered to us:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Localhost is exempt, so development tests don't stop because they hit the limit. Otherwise you'd lock yourself out of your own machine.

IPs can be simulated in development with X-Forwarded-For and a DETECTOR_DEV=1 variable. That way you can test the limit without deploying.

In production, DETECTOR_TRUST_PROXY=1 to read the real IP behind a proxy. Otherwise all users would share the proxy's IP.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;The limit is simple, but it's the difference between opening deep mode with a safety net and opening it without one.&lt;br&gt;
The Four Test Files, and What They Actually Tell Us&lt;/p&gt;

&lt;p&gt;To measure without fooling ourselves, we wrote four files:&lt;br&gt;
File    What it measures    Result&lt;br&gt;
test_debe_detectar.py   Recall (catch the bad stuff)    14–15 findings, almost all correct&lt;br&gt;
test_no_debe_detectar.py    Precision (don't flag the good stuff)   0 findings ✅&lt;br&gt;
test_limitaciones.py    Known false negatives/positives 1 finding (a true positive by heuristic)&lt;br&gt;
test_syntax_error.py    Broken syntax   0, "file looks clean" ❌&lt;/p&gt;

&lt;p&gt;The good part: precision is perfect on clean code. The honest part: there's a UX bug with broken syntax (the detector says "clean" when the file doesn't even compile) and there are structural false negatives (dynamic imports, from-import of names, nonexistent submodules) that require deeper analysis. We have them documented, not hidden.&lt;br&gt;
What's Next&lt;/p&gt;

&lt;p&gt;So the project is no longer "an API registry." It's a detector that actually verifies against PyPI, with caching, opt-in deep mode, and usage control. What's left to review from this milestone:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Wheel download events in the progress bar (today, with deep mode, the bar freezes while the wheel downloads).

--deep in the CLI, so the terminal has the same power as the web.

Reporting syntax-error as a finding, instead of saying "file looks clean".

Evidence and status on each finding (confirmed, probable, unverifiable).

pyright/mypy only as a local option, never on the server (slow, requires installation, and without stubs it flags nothing).
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;And there's one more thing we'd like to mention. Once we have hosting — complicated from my country under siege — we'll launch a web app that already allows running the analyses online. It's a tool that's already built and working locally: you upload a .py, you see progress in real time (with an SSE progress bar), and you get the findings, including deep mode with verification against PyPI wheels. But for that web app to be sustainable and not fall over at the first sign of traffic, we need a contribution to the tool's development. It's not a payment to use the service: it's a way for the project to cover hosting, the bandwidth for wheel downloads, and maintenance time. The idea is that anyone who wants to use deep mode intensively can contribute, and anyone who just wants to try basic analysis can do so without barriers. The code will remain open; the hosted web app will be how the project sustains itself. We'll share more details when it's ready.&lt;/p&gt;

&lt;p&gt;The previous article ended with an idea: "the linter sees what it knows; the type checker sees what's in the signature; neither knows whether the method actually exists." This is an attempt to close that gap. Not with a bigger list, but by reading what the package publishes. It's still not perfect: there are false negatives we acknowledge and a syntax bug pending. But the difference between "there's no hallucination" and "it couldn't be verified" is no longer hidden: we measure it, we document it, and we try to reduce it.&lt;/p&gt;

&lt;p&gt;If you try deep mode and find a false positive or one that slips through, I'd love to hear about it. This project gets corrected by what others find.&lt;br&gt;
References&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Spracklen, J., Wijewickrama, R., Sakib, A.H.M.N., Maiti, A., &amp;amp; Viswanath, B. (2025). We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. 34th USENIX Security Symposium. arXiv:2406.10279.

Miranda-Pena, C., Reeson, A., Paris, C., Poon, J., &amp;amp; Kummerfeld, J.K. (2026). An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations. arXiv:2604.07755.

Tileria, M., et al. (2026). Hallucination Inspector: A Fact-Checking Judge for API Migration. arXiv:2604.20202.

Raj, A., &amp;amp; (co-author). (2026). Names Can Hurt: Spotting Slopsquatting Risks Caused by Package Name Hallucinations in Local Coding LLMs. arXiv:2608.23897.

Churilov, A. (2026). The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort. arXiv:2605.17062.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>python</category>
      <category>testing</category>
    </item>
    <item>
      <title>Why API Hallucinations Are Invisible to Your Linter (And How I Built a Parser to Catch Them.</title>
      <dc:creator>Carlos Luque Zayas Bazan</dc:creator>
      <pubDate>Mon, 26 Oct 2026 21:31:01 +0000</pubDate>
      <link>https://dev.to/carlos_luquezayasbazan_/why-api-hallucinations-are-invisible-to-your-linter-and-how-i-built-a-parser-to-catch-them-1185</link>
      <guid>https://dev.to/carlos_luquezayasbazan_/why-api-hallucinations-are-invisible-to-your-linter-and-how-i-built-a-parser-to-catch-them-1185</guid>
      <description>&lt;p&gt;When an AI coding assistant writes requests.get_json() instead of response.json(), the code compiles. Tests pass. Type checkers don't complain. SAST tools don't flag it. And yet the method doesn't exist.&lt;br&gt;
This is an API hallucination. It's a specific failure mode of AI-generated code that most static analysis tools miss because they're looking for known vulnerabilities or suspicious patterns, not for calls to functions that were never real to begin with.&lt;br&gt;
I've been building a CLI in Go that parses Python code and checks whether the functions and methods it calls actually exist in the library. The core idea is simple: if you know the valid API surface of requests, json, and os, you can flag requests.get_json() because get_json isn't in that surface.&lt;br&gt;
The parsing is done with gotreesitter, a pure-Go implementation of tree-sitter. I chose it over the CGO binding because I needed to compile on Windows without MinGW. The tradeoff is performance (the pure-Go version is slower), but for a CLI that analyzes individual files, the difference is milliseconds.&lt;br&gt;
The hard part isn't the parser. It's the registry of valid APIs. For the first version, I hardcoded the valid methods for three libraries. That's enough to demonstrate the concept, but it doesn't scale. Expanding it to pandas, numpy, or django requires either manual curation or extracting method names from type stubs.&lt;br&gt;
What I've learned so far: API hallucinations are predictable. They follow patterns — plausible-sounding method names, slight variations on real APIs, invented functions that "should" exist. A deterministic checker can catch them if it knows what's real. The question is how to build that "what's real" database without drowning in maintenance.&lt;br&gt;
The code isn't public yet, but I'm sharing the approach because I think the problem is worth solving. If you've seen API hallucinations in your own AI-generated code, I'd like to hear how you catch them.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>api</category>
      <category>cli</category>
      <category>python</category>
    </item>
  </channel>
</rss>
