Two weeks ago, a staging deployment crashed with an error that made no sense. The code called pandas.compat.numpy.func and expected a utility that has not existed since version 0.25. The AI wrote the call because it had seen a similar pattern in an old tutorial, and nobody questioned it. We fixed the line in seconds, but the real problem took longer to admit: every AI-generated file was a rumor mill of plausible imports.
AI coding assistants are brilliant at producing code that looks right, but they do not install a package before suggesting its API. They sample from patterns in training data, and those patterns include outdated, internal, or entirely invented functions. The classic debugging tools do not help because nothing is wrong until a code path actually executes the phantom call. Static type checkers only catch errors when they know the library, which for long-tail dependencies they rarely do.
Instead of waiting for runtime, we can ask a language model directly whether each imported API exists, and treat the answer as a first-pass filter. The query is short, the context is tiny, and the batch fits inside a free tier easily. For this experiment I used MonkeyCode's free models for the questions and a free server to schedule the recheck, because the whole workflow costs zero dollars and zero committed compute. Free tiers change, so verify the current limits, but the pattern is what matters. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The Oracle in One Screen
The script is ninety lines and reads a file, extracts every import and attribute reference, then asks the model to label each one as 'exists', 'missing', or 'unknown'. The prompt forces the model to commit: it must answer with one of those words and a one-line reason. This prevents the evasive 'I cannot be sure' responses that make automation impossible.
Here is the core prompt template:
System: You are an API oracle. You know Python libraries up to your training cutoff. For each symbol, reply EXIST, MISSING, or UNKNOWN. Then one short proof sentence. Do not hedge.
Each request carries the symbol, the package, and the version we are using. The version matters more than people expect, because many 'AI hallucinations' are actually correct APIs from older releases.
The Code
Here is the full script, which you can save as api_oracle.py.
#!/usr/bin/env python3
"""api_oracle.py - check every imported API against a model's knowledge."""
import ast
import json
import os
import sys
import time
from pathlib import Path
import requests
LLM_URL = os.environ.get('LLM_URL')
LLM_KEY = os.environ.get('LLM_KEY')
MODEL = os.environ.get('LLM_MODEL')
LABELS = ('EXIST', 'MISSING', 'UNKNOWN')
PROMPT = '''You are an API oracle. You know Python libraries up to your training cutoff.
For each symbol, reply with one of {labels}, then a short proof sentence.
Do not hedge. Only answer from your knowledge.
Package: {{package}}
Version: {{version}}
Symbol: {{symbol}}
'''.format(labels='/'.join(LABELS))
def extract_symbols(path):
tree = ast.parse(Path(path).read_text(encoding='utf-8'))
symbols = []
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for alias in node.names:
symbols.append(('import', alias.name, None))
elif isinstance(node, ast.ImportFrom):
for alias in node.names:
symbols.append(('from', node.module, alias.name))
elif isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
if isinstance(node.func.value, ast.Name):
symbols.append(('attr', node.func.value.id, node.func.attr))
return symbols
def ask_model(package, version, symbol):
response = requests.post(
LLM_URL,
headers={'Authorization': f'Bearer {LLM_KEY}'},
json={
'model': MODEL,
'messages': [
{'role': 'system', 'content': PROMPT},
{'role': 'user', 'content': json.dumps({
'package': package,
'version': version,
'symbol': symbol,
})}
],
'temperature': 0.0,
},
timeout=30,
)
response.raise_for_status()
text = response.json()['choices'][0]['message']['content'].strip()
label = next((lab for lab in LABELS if text.startswith(lab)), 'UNKNOWN')
return label, text
def main():
if len(sys.argv) != 2:
print('Usage: api_oracle.py <source.py>')
sys.exit(2)
symbols = extract_symbols(sys.argv[1])
if not symbols:
print('No symbols found.')
sys.exit(0)
report = []
for kind, package, attr in symbols:
symbol = attr or package
label, proof = ask_model(package, 'installed', symbol)
report.append({'kind': kind, 'package': package, 'symbol': symbol, 'label': label, 'proof': proof})
if attr:
print(f'{label:>8} {package}.{attr}')
else:
print(f'{label:>8} {package}')
time.sleep(0.3)
Path('oracle-report.json').write_text(json.dumps(report, indent=2))
missing = [r for r in report if r['label'] == 'MISSING']
print('Missing:', len(missing))
if missing:
sys.exit(1)
if __name__ == '__main__':
main()
Run it with environment variables pointing at your LLM endpoint.
export LLM_URL="https://api.monkeycode.example/v1/chat/completions"
export LLM_KEY="your-key"
export LLM_MODEL="some-free-model-name"
python api_oracle.py src/main.py
The script is intentionally crude. It does not resolve relative imports or understand aliases perfectly; it is a tripwire, not a compiler. The important part is that it forces the model to take a position, which makes the output actionable.
A Sample Run
Consider a file with these four imports: import pandas, from sklearn.metrics import confusion_matrix, import internal_tools, and from legacy_utils import transform. A run might produce this output.
EXIST import pandas
EXIST sklearn.metrics.confusion_matrix
UNKNOWN import internal_tools
MISSING legacy_utils.transform
The last line is the one that deserves a human. legacy_utils may be an internal package that the model simply never saw, or it may be a real library from before the training cutoff. Either way, a one-second check is cheaper than a 3 a.m. page.
How to Read the Verdicts
| Label | Meaning | Action |
|---|---|---|
| EXIST | The model has seen this API and asserts it exists | Keep, but verify with one real import test |
| MISSING | The model cannot find it and says so | Investigate manually before merging |
| UNKNOWN | The model refuses to commit | Add a unit test that exercises the symbol |
That table is the real deliverable. Without it, the output is just a wall of text. With it, a junior dev can triage AI-generated code in seconds.
Free Server as a Nightly Gate
Rather than running this ad hoc, I put the oracle on a free server plan and set a cron job to run it over the newest pull request each morning. The free server is enough for a single scheduled job that takes under a minute, and the report lands in a file for the team to read.
0 8 * * * cd /srv/oracle && export LLM_URL=... && export LLM_KEY=... && export LLM_MODEL=... && python api_oracle.py $(ls -t /tmp/newest*.py | head -1) >> oracle.log 2>&1
The schedule turns a clever trick into a habit. The team no longer asks whether the AI wrote real code; they ask where the oracle failed.
Limitations
First, the model is frozen at its training cutoff. A library released last month will be mislabeled as MISSING, which produces false alarms. Version pinning helps only if your model knows that version. Second, internal packages simply do not exist in training data; the oracle will say UNKNOWN for every one of them. Third, the extraction is shallow, so import pandas as pd is treated as the symbol pandas, not every pd. usage. The script catches the import error case, not all attribute misuse.
Also, the model itself can hallucinate even when told to hedge. That is why the prompt demands a proof sentence, which a human can skim. Run this on a small sample before trusting it on a large one.
Who Should Skip This
Teams that already own a full dependency scanner with a live package index should not switch. Teams that only use a few well-known libraries will waste time on false alarms. And anyone who wants a guarantee should remember that this oracle is a heuristic, not a proof.
If the risk is worth it, the next step is easy: write a quick script that collects all imports from your current service and feed them to the free models. You will likely find one ghost API, which is exactly the point. When you do, you can thank me later by deleting the line that caused the crash.
Top comments (0)