Implementation Scout: Making AI Prove Its Coding Advice Instead of Just Sounding Confident

TL;DR: Built a skill that forces Claude/ChatGPT to fetch real GitHub source before giving implementation advice, instead of confidently pattern-matching an answer. It works. It also has one specific, confirmed failure mode I couldn't fully close — and that failure mode turned out to be more interesting than the parts that worked.

The problem

Ask an LLM "how do I implement X" and you get a confident, well-formatted answer every single time — whether the model actually knows, half-remembers, or is pattern-matching from something adjacent.

Here's the actual problem: right and wrong look identical. Same tone, same confidence, same polish.

So I built Implementation Scout — a skill for Claude, ChatGPT, and anything else supporting the Agent Skills format. The rule it enforces:

Go find the real upstream implementation. Fetch it. Prove what you can prove. Say "not verified" for everything you can't.

Repo: github.com/shouvik12/implementation-scout

What it does

Reconstruct the workflow before searching. Not "what tech was mentioned" — what's the actual failure point. This alone catches a lot: "make Keycloak B trust Keycloak A" can get answered with a correct-looking but wrong-layer OIDC login pattern if you search on keywords instead of the real workflow.
Fetch real source. Every time. Not "here's generally how this works" — an actual web_fetch against a real file, this run.
Pin the version. Tagged release over floating branch, always, and say so when it can't.
Investigate provenance separately from source. A file existing in a repo ≠ proof of which commit introduced the logic you're citing. This is the rule that turned out to matter most.
Six evidence axes, scored independently: SOURCE, PROVENANCE, VERSION, RUNTIME PATH, INTEGRATION POINT, APPLICABILITY. A strong result on one never pulls another toward verified.
CHANGE HERE / DO NOT CHANGE / DO NOT COPY — not a code dump. Naming what not to touch caught more real mistakes than anything else in testing.
Stop when evidence stops. NOT VERIFIED is a win condition, not a failure.

A run that actually impressed me

Asked it how nginx's least_conn load balancer picks an upstream server. Here's the part that mattered:

SOURCE: VERIFIED
  Mirror actually fetched: docs.rs crate "nginx-src" — used because
  direct github.com/nginx/nginx blob URLs weren't fetchable without a
  prior search hit on that exact URL. docs.rs was reachable instead.

VERSION: nginx-src crate version 1.28.5+1.28.3 — this is the version
  actually fetched, NOT independently cross-checked against a
  github.com/nginx/nginx tag. Treat as "matches nginx ~1.28.x," not a
  commit-exact pin.

I independently verified this — real file, real function, real version string, down to the exact docs.rs page.

But that's not the impressive part. The impressive part: it hit a genuine wall (couldn't reach the canonical repo), switched to a mirror, and told me exactly how that degrades the precision of its own claim. Not "here's the answer" — "here's the answer, and here's exactly how confident you should be in it, and why."

That's the actual product. Not answers. Calibrated answers.

The failure that actually matters

I stress-tested this across Keycloak, Grafana, PostgreSQL, Stripe, Kafka, and Cassandra — unrelated domains, across ChatGPT, Gemini, and Claude.

The failure wasn't bad citations. It was worse:

Models marking evidence axes VERIFIED without completing the verification chain. Sounding evidence-based without actually finishing the work.

Sharpest on PROVENANCE and INTEGRATION POINT. A commit existing isn't provenance — you have to open the diff and confirm it actually introduced what you're citing. "This pattern is documented" isn't evidence it applies to your system.

One clean example: asked for idempotent Stripe webhook handling with zero details about my stack. Correct behavior: IMPLEMENTATION: BLOCKED. What I got instead: a confident IMPLEMENTATION: READY, with RUNTIME PATH, INTEGRATION POINT, and APPLICABILITY all marked VERIFIED — justified by describing the general Stripe pattern, not anything about my actual system.

I threw a lot at this:

A blocking completion checklist that has to pass before any report ships
A rule requiring every VERIFIED claim to point to a reconstructable EVIDENCE ID — real tool output, not reasoning
Inline reminders placed directly at the point of failure in the report template, not just stated once upstream

Each one helped. None fully closed it, on every model.

ChatGPT was consistently the most disciplined here. Best moment: it found a plausible-looking commit, opened the actual diff, saw it didn't introduce the cited logic, and rejected its own citation instead of using it anyway. That rule — diff context is not provenance — is now one of the most heavily enforced parts of the skill, precisely because it's the hardest thing to get a model to police in itself.

My honest conclusion: this isn't fully fixable from inside a prompt. It's a confidence-calibration tendency, not a wording gap. The durable fix is probably a system that inspects the actual tool-call transcript and mechanically checks "did a real fetch happen" — instead of trusting the model's self-report. That's a bigger, separate project. Haven't built it yet.

What shipped anyway

The full pipeline above
A documented, un-hidden known limitation
A short guide showing what real verification looks like next to confident fakery:

# real
SOURCE: VERIFIED
Repository: keycloak/keycloak
File: services/.../AbstractTokenExchangeProvider.java
Commit: 3a72090... (or a tagged release)
→ you can open that link. it's real.

# fake, but reads identically confident
SOURCE: VERIFIED
Cassandra's bootstrap process supports nodetool bootstrap resume...
→ no repo. no file. no commit. no link.

The one rule to actually use this safely: before acting on anything marked VERIFIED or IMPLEMENTATION: READY, ask — "what's the actual URL or commit you fetched this from?" If it can't produce one, that gap is the real signal. Not the label.

Try to break it

MIT licensed, open, actively looking for failure modes I haven't found yet.

github.com/shouvik12/implementation-scout

If you run it against your own stack and it either impresses you or confidently makes something up — I want to hear which.

DEV Community