DEV Community

Cover image for Your repo has a file called stripe/webhook.ts. It does nothing.
Aviad Shakargy
Aviad Shakargy

Posted on

Your repo has a file called stripe/webhook.ts. It does nothing.

#ai #opensource #python #tooling

Here is a file from a real, well-known open-source repository:

pages/api/stripe/webhook.ts
Enter fullscreen mode Exit fullscreen mode

If you grep a codebase and find that file, you will conclude the app handles Stripe webhooks. Your coding agent will conclude the same thing, faster, and with more confidence.

Here is the entire handler:

export default function handler(_req: NextApiRequest, res: NextApiResponse) {
  res.status(404).json({ message: "Billing webhooks are not available in community edition" });
}
Enter fullscreen mode Exit fullscreen mode

It does nothing. The feature lives in a paid edition. The file name makes a claim the code does not keep.

I found this while testing my own tool against Cal.com, and it changed what I was building.

DevTime started as a local-first CLI that scans a repository and explains it from evidence. As of v0.4.0, it has become something more specific: a verification layer. You state a claim about the repository, and DevTime tells you whether the code can actually back it.

The Idea: Claims, Not Vibes

A claim is a statement about a repository. "Billing webhooks verify signatures." "Authentication uses JWT access tokens."

Most tools answer these questions with pattern matching and confidence. DevTime answers with a status and receipts:

pipx install devtime-ei
cd your-repo
dtc init
dtc scan
dtc verify billing-webhook-signature
Enter fullscreen mode Exit fullscreen mode 
Billing Webhook Signature Verification
Claim: Incoming billing webhooks verify the provider's signature.
Status: SUPPORTED

Why:
  - Signature verification behavior evidence was found.
  - A test exercises signature behavior.

Supporting evidence:
  - src/billing/stripe-webhook.ts  [strong]
      Verifies the provider's webhook signature.
  - tests/stripe-signature.test.ts  [moderate]
      Test exercises webhook signature behavior.
Enter fullscreen mode Exit fullscreen mode

Four possible statuses:

  • SUPPORTED - required behavior evidence exists in the current scan
  • WEAK - the surface exists, but the proving evidence is missing
  • CONTRADICTED - credible evidence conflicts with the claim
  • UNKNOWN - no relevant surface found, or coverage cannot responsibly decide

Statuses mean "per the evidence rules", never formal proof. The tool says so itself.

Contradictions Show Both Sides, Always

Back to that stub file. Run the same claim against a repo where the handler is a 404 stub:

Status: CONTRADICTED

Contradictions:
  - The billing webhook endpoint cannot verify signatures
    because it is a disabled stub.
      claimed:  pages/api/stripe/webhook.ts is named and routed
                as a billing webhook endpoint.
      observed: The handler's only behavior is a 404/501 response.
Enter fullscreen mode Exit fullscreen mode

No vague "authentication may be inconsistent" warnings. A contradiction names what claimed the behavior, what the code actually does, and the exact paths. You can go check.

There is a second detector for the classic case: documentation says JWT, but the only JWT usage in the code is invitation tokens, not access tokens. Both sides, exact files.

One rule I care about a lot: absence is not a contradiction. Documentation with no matching code is WEAK, missing evidence. CONTRADICTED requires positive conflicting evidence on both sides. A tool that cries contradiction on every gap would be noise within a week.

Truth and Freshness Are Separate Axes

A claim you verified last month is not the same as a claim that still holds.

Every verification stores fingerprints of its evidence files. When any of them changes, the claim goes stale, and DevTime names the file:

billing-webhook-signature
  last status: SUPPORTED   freshness: STALE
    changed since verification: src/billing/stripe-webhook.ts
Enter fullscreen mode Exit fullscreen mode

Only evidence files count. You can rewrite the rest of the repo and nothing flags. That restraint is deliberate: staleness alerts that fire on every commit train people to ignore staleness alerts.

The diff review picks this up too:

dtc risk --diff
Enter fullscreen mode Exit fullscreen mode 
Claim impact:
  - billing-webhook-signature (previous status: SUPPORTED)
      changed evidence: src/billing/stripe-webhook.ts
      re-verify: dtc verify billing-webhook-signature
Enter fullscreen mode Exit fullscreen mode

Your diff does not just touch files. It destabilizes specific verified claims, and now the review says which ones.

Why This Matters More With Agents

Coding agents read file names and README statements, then act on them with full confidence. The stub webhook above is exactly the kind of thing an agent gets wrong: it would "know" the repo handles Stripe webhooks, and build on payment logic that does not exist in that edition.

DevTime ships a read-only MCP server, so agents can ask instead of guess:

pipx install "devtime-ei[mcp]"
claude mcp add devtime -- dtc mcp start
Enter fullscreen mode Exit fullscreen mode

The agent gets four tools, including this one:

verify_claim -> status, why, supporting evidence, contradictions,
                missing evidence, coverage limitations
Enter fullscreen mode Exit fullscreen mode

Local stdio only. No network listener. No source code returned. And no LLM anywhere in the truth path: verification is deterministic and rule-driven, so the same repo state always produces the same answer.

What It Is Not

Honesty section, because the tool is built on one:

  • It is a heuristic scanner. Evidence comes from static patterns, not execution.
  • Two built-in claims so far (billing webhook signatures, JWT authentication). User-defined claims are deliberately postponed until the built-in ones prove trustworthy on real repositories.
  • Statuses are evidence policy, not security certification.
  • Strongest on TypeScript/Next.js/Express/FastAPI-style repos. Coverage limits are documented, and the tool prints its own blind spots during scans.

Every wrong output can become a regression fixture. That is the deal I offer: if DevTime says something your repository cannot back up, that report makes the tool permanently better.

Try It 

pipx install devtime-ei
dtc demo init
cd devtime-demo-saas
dtc init
dtc scan
dtc verify
Enter fullscreen mode Exit fullscreen mode

The demo repo ships in the package, so you can see SUPPORTED and then break it yourself and watch the status flip.

Links

GitHub logo Shakargy / devtime

Local-first Engineering Intelligence for software repositories.

DevTime

Local-first Engineering Intelligence for software repositories.

DevTime helps a codebase explain itself from evidence.

It scans code, tests, configs, routes, and decisions to identify supported software concepts, link claims to files, surface uncertainty, and warn about a narrow set of risky changes.

No cloud. No telemetry. No code execution. No AI required.

DevTime terminal demo - install, scan, and explain a repo from evidence

Prefer video? Watch the 2-minute demo: DevTime scans a repo locally, explains concepts from evidence, surfaces uncertainty, catches a risky diff, and shows how a corroborated decision improves understanding.

Try DevTime in 60 seconds

pipx install devtime-ei
dtc demo init
cd devtime-demo-saas
dtc init
dtc scan
dtc concepts
dtc explain "Billing Webhooks"
Enter fullscreen mode Exit fullscreen mode

The PyPI distribution is devtime-ei. The Python package remains devtime, and the CLI command remains dtc. dtc demo init copies a small static example repo into ./devtime-demo-saas so you can try DevTime without cloning this repository.

From source

git
…
Enter fullscreen mode Exit fullscreen mode
View on GitHub

If you run it on a repo and it claims something your code cannot prove, open an issue. That is exactly the feedback I am looking for!

Top comments (2)

Collapse
 
topstar_ai profile image
Luis Cruz •

I was particularly intrigued by the concept of "claims" vs "vibes" in verifying repository behavior, and how DevTime provides a verification layer to confirm these claims. The example of the Stripe webhook handler returning a 404 response, despite the file name suggesting otherwise, highlights the importance of verifying claims through evidence. I appreciate how DevTime provides a clear status and receipts for each claim, such as SUPPORTED, WEAK, CONTRADICTED, or UNKNOWN, giving developers a clear understanding of their repository's behavior. How do you envision DevTime handling more complex claims that involve multiple interconnected components or services?

Collapse
 
komo profile image
Reid Marlow •

This is the failure mode I keep coming back to with coding agents: file paths are strong claims, but repos are full of decoys, stubs, and edition boundaries.

I like the claim-first shape here. The hard part is deciding what counts as enough evidence for a negative answer. Do you require an executable path/test, or is static reachability usually enough for DevTime?