You review a PR, the code calls client.batchUpdateWithRetry(), it reads naturally, fits the surrounding code's style, and you move on. Later, the build fails, or worse, it doesn't fail because a linter or type checker didn't catch it, and someone discovers at runtime that no such method exists on that client. The AI assistant that generated it wasn't being careless. It was doing exactly what it's built to do, and that's worth understanding to know where to watch for it.
Why This Happens at All
Large language models generate code by predicting what token is statistically likely to come next, given everything that came before, including the surrounding code, the library being used, and patterns from a huge volume of training data covering how that library (or libraries like it) are typically used. For extremely common libraries with widespread, consistent usage patterns in training data, this produces remarkably accurate output most of the time. The failure shows up specifically when a library is less common, has gone through breaking API changes across versions, or when the model is filling in a gap with the most statistically plausible completion rather than a verified fact.
The result reads as confident because the model isn't distinguishing between "I've seen this exact method used this way many times" and "this is the most plausible-sounding completion given the pattern." Both produce equally fluent, equally confidently formatted output. Nothing about the code's surface signals which case you're in.
Where This Shows Up Most
A few situations raise the odds of this particular failure mode meaningfully:
Less popular or newer libraries. A library with a smaller footprint in public training data gives the model less signal to draw on, and it's more likely to blend patterns from similar-but-different libraries into a plausible-sounding but incorrect API surface.
Libraries that changed their API across major versions. If a library renamed a method, changed a parameter's type, or deprecated an option between versions, a model trained on a mix of old and new documentation and code examples can generate a call that was valid in an older version but not the one actually installed, or vice versa.
Internal or private libraries. A model has no training exposure at all to your team's internal libraries unless that context is provided directly. Any call to an internal API is either grounded in what's visible in the current codebase and prompt, or it's a plausible-sounding guess extrapolated from how a similar public library would typically work.
Configuration options and parameter names specifically. Method names are relatively easy to verify by checking whether the call even resolves. Individual parameter names and configuration keys are easier to get subtly wrong in a way that still type-checks or compiles if the language and framework are permissive about extra or misnamed options, silently ignoring a config value instead of erroring on it.
Why This Is Hard to Catch by Reading Alone
The core problem is that fabricated API calls are indistinguishable from correct ones by tone or structure. A hallucinated method name follows the exact same naming conventions as real methods in that library, because the model learned those conventions from real usage. Nothing about the code itself signals "verify this one specifically." That's precisely why this failure mode survives a normal skim-and-approve review pass, and why it needs a specific, deliberate countermeasure rather than general carefulness.
What Actually Catches It
Type checking and compilation are the first, cheapest filter. In a statically typed language, a call to a genuinely nonexistent method fails to compile, which catches the most blatant cases for free. This doesn't cover every case, since a permissive dynamic language, or a dynamically dispatched call, can silently accept a nonexistent method call until it's actually invoked at runtime.
Tracing the call against real documentation, not memory, is the reliable manual check. If a reviewer is even slightly unsure whether a specific method, parameter, or config key genuinely exists as written, opening the actual current documentation for the installed version takes under a minute and resolves the question definitively, versus trusting that it looks right.
Integration tests that actually exercise the call path catch what static analysis misses. A test that mocks the client entirely never discovers that a method doesn't exist on the real object; a test that exercises the real client (against a sandbox or test environment where feasible) will fail immediately if the call is fabricated.
Pinning and reading the exact installed version matters more with AI-generated code than it used to. If a codebase has multiple versions of a library referenced across its history, or the assistant's training data spans several versions, explicitly telling the assistant which version is installed, and verifying generated calls against that specific version's docs, closes a meaningful chunk of this gap.
A Related Case: Correct Method, Wrong Assumptions About Behavior
A quieter variant of this same problem is a method call that genuinely exists and compiles fine, but where the generated code makes an incorrect assumption about its behavior, whether it's synchronous or async, whether it throws or returns an error value, whether it mutates its input or returns a new copy. This doesn't fail to compile in most languages, and it doesn't always fail tests either, if the tests were generated from the same incorrect assumption. It shows up as a subtle behavioral bug, often around error handling or ordering, that's harder to catch than a flatly nonexistent method because the code runs without obvious complaint most of the time.
The same countermeasure applies here as with fabricated calls: when a reviewer isn't certain about a method's exact behavior, particularly around error handling and async semantics, checking the real documentation takes less time than debugging the resulting production issue later. This is worth treating as a habit specifically for any call the reviewer hasn't personally used many times before, regardless of whether the method itself is real.
Why Newer Codebases and Fast-Moving Libraries Carry More Risk
Libraries under active development, shipping breaking changes across minor or major versions frequently, are a specific risk multiplier here. A model's training data reflects a snapshot in time, and a library that's changed its API meaningfully since that snapshot leaves a gap between what the model confidently generates and what the currently installed version actually supports. Teams working with fast-moving frameworks or in-house libraries under active iteration should treat generated calls into those specific libraries with more scrutiny than calls into a stable, slow-changing dependency with a long, consistent history.
This Is One Piece of a Larger Review Problem
Fabricated API calls are one specific, well-documented failure mode among several that make AI-generated code need a different review approach than human-written code, alongside plausible-but-wrong logic and silently missing context a prompt didn't provide. Our broader guide on reviewing AI-generated code without rubber-stamping it covers the full set of failure modes and a review checklist built around them.
137Foundry helps engineering teams build review and tooling practices tuned to how AI-generated code actually fails, not how human-written code fails. See our full services or read more about how we work.
For more on the underlying model behavior, Wikipedia's entry on hallucination in AI systems covers the broader phenomenon this is a specific instance of, and most language model providers, including Anthropic, publish their own guidance on where model-generated output needs independent verification rather than direct trust. MDN's web documentation and equivalent official docs for whatever library you're using remain the fastest, most reliable way to verify a specific method or parameter actually exists as generated, faster in practice than trying to reason about whether it looks plausible.
Top comments (0)