DEV Community

hao li
hao li

Posted on

Your LLM writes APIs that don't exist. Ruff can't tell. This can.

#opensource #testing #python #ai

Your LLM writes APIs that don't exist. Ruff can't tell. This can.

Measured studies put deprecated-API usage in LLM-generated code at 25–38% across 7 models. That's not a typo problem — it's a verification problem. Your linter checks your code against a fixed list of rules. It cannot answer the two questions that actually matter when the code was written by a model:

  1. Does this API actually exist? A model can invent requests.post_json — perfectly plausible, completely fictional. No rule list contains it, so ruff stays silent.
  2. Is this real API already deprecated? Deprecation is a property of the installed library, not of your source text. A static rule engine would need a perpetually updated database of every library's deprecations.

I built a small CLI that answers both — by checking for real.

apimirage

apimirage parses your file with ast, collects every call made through an imported name, then verifies each one with a genuine importlib import and inspect.signature. Nothing is ever executed — imports are only used for existence checks.

Zero dependencies. Python standard library only.

pip install apimirage
Enter fullscreen mode Exit fullscreen mode

Scan a file:

$ apimirage app.py
app.py:12: [ghost-api] requests.post_json -- attribute does not exist on the real module -- possibly hallucinated
app.py:27: [ghost-param] os.path.join(pretty=...) -- keyword argument 'pretty' is not accepted by the real signature
app.py:41: [deprecated] mymod.old_fn -- API is deprecated: docstring says: Deprecated since 2.0: use new_fn() instead.
apimirage: 3 findings in 1 file
Enter fullscreen mode Exit fullscreen mode

Exit codes are CI-friendly: 0 = clean, 1 = findings, 2 = usage error. There's a --format json mode for piping into other tools:

$ apimirage -f json -q app.py | head -20
[
  {
    "kind": "ghost-api",
    "file": "app.py",
    "lineno": 12,
    "name": "requests.post_json",
    "detail": "attribute does not exist on the real module -- possibly hallucinated"
  },
  ...
]
Enter fullscreen mode Exit fullscreen mode

What it catches

kind meaning
ghost-module the imported module can't be imported at all — likely hallucinated
ghost-api the module is real, but the attribute doesn't exist on it
ghost-param the function is real, but the kwarg isn't in its real signature
deprecated the API advertises deprecation (__deprecated__ per PEP 702, or a deprecation docstring)
unverifiable the import itself errored, so existence couldn't be confirmed

It also works as a library:

from apimirage import check_file

for finding in check_file("app.py").findings:
    print(finding.one_line())
Enter fullscreen mode Exit fullscreen mode

Why not just ruff / pylint?

Ruff and pylint are static rule engines: they match your code against patterns someone wrote down in advance. That works for style and for APIs they already know about. It fundamentally can't catch a hallucination, because a hallucination is novel by definition — no rule list contains requests.post_json.

apimirage doesn't consult a rule list at all. It imports the actual module you imported and asks: does this attribute exist? Does this signature accept this kwarg? Does this object say it's deprecated? The ground truth is your installed environment, not a database.

The honest trade-offs: it only verifies names statically resolvable through imports (dynamic getattr and monkeypatching are invisible to it), and importing a module does run its top-level code — it never calls anything, but heavy import-time side effects will run. An API that exists in your installed version but not your deploy target is judged against what's installed here.

Where this fits

If you generate code with LLMs — your own agents, CI pipelines, eval harnesses — apimirage is a cheap gate between "the model wrote it" and "we ship it". Fifteen stdlib-only tests, MIT licensed.

Top comments (0)