DEV Community

Cover image for I scanned Cal.com with my evidence-based CLI. The best result was what it refused to say
Aviad Shakargy
Aviad Shakargy

Posted on

I scanned Cal.com with my evidence-based CLI. The best result was what it refused to say

#ai #programming #python #opensource

I pointed my scanner at Cal.com this week, fully expecting to write a post about everything it got wrong.

The most interesting result was the thing it refused to say.

Quick context: I build DevTime, a local-first CLI that scans a repository and helps it explain itself from evidence. It detects a closed set of concepts (authentication, billing webhooks, background jobs, data export, admin permissions, file uploads), links every claim to evidence files, and reports uncertainty when it cannot prove something. No cloud, no telemetry, no code execution during scan, no AI.

The Setup

Cal.com is a well-known open-source scheduling platform. Big TypeScript monorepo, Next.js frontend, NestJS API, Stripe integration. If you wanted to stress a heuristic scanner, this is a good place to do it.

The run:

pipx install devtime-ei
git clone --depth 1 https://github.com/calcom/cal.com
cd cal.com
dtc init
dtc scan
Enter fullscreen mode Exit fullscreen mode

The scan:

Scan complete. Scanned 5736 files, 24516 signals, 3 concepts in 4.2s.
Enter fullscreen mode Exit fullscreen mode

Local, 4.2 seconds, nothing left the machine.

What It Found 

1. Authentication      confidence: high   debt: medium
2. Billing Webhooks    confidence: low    debt: high
3. Background Jobs     confidence: low    debt: high
Enter fullscreen mode Exit fullscreen mode

Authentication was solid. Real evidence, real routes:

- Authentication is present and supported by behavior evidence.
  confidence: 0.82
  evidence: apps/web/app/api/auth/forgot-password/route.ts,
            apps/web/app/api/auth/oauth/me/route.ts
Enter fullscreen mode Exit fullscreen mode

But Billing Webhooks at LOW confidence? Cal.com is a scheduling SaaS with Stripe integration. Of course it has billing webhooks. I assumed I had found my first miss.

The Refusal

Here is what DevTime actually said:

- Possible Billing Webhooks signals detected. Billing Webhooks behavior
  is not established from current evidence.
  confidence: 0.45

Uncertainty:
  - Only dependency or manifest evidence was found for Billing Webhooks;
    presence is not confirmed behavior.
Enter fullscreen mode Exit fullscreen mode

So I went to check manually. The repo has exactly the file you would expect:

apps/web/pages/api/stripe/webhook.ts
Enter fullscreen mode Exit fullscreen mode

Case closed, right? A file named stripe/webhook.ts in a Stripe-integrated SaaS. Any developer grepping the repo would say "here is their webhook handler". Any AI agent reading file paths would state it as fact.

Here is the entire handler:

export default function handler(_req: NextApiRequest, res: NextApiResponse) {
  res.status(404).json({ message: "Billing webhooks are not available in community edition" });
}
Enter fullscreen mode Exit fullscreen mode

It is a stub. The open-source snapshot of Cal.com does not process billing webhooks. That behavior lives in their enterprise edition. The second webhook route in the repo is the same story.

DevTime could not find behavior evidence because there is no behavior to find. The refusal was the correct answer.

That is the whole thesis of the tool in one example. A confident story built from file names would have been wrong. The repository could not prove billing webhooks, so the scanner said so.

Where It Was Actually Wrong

I promised misses, and there are real ones.

Background Jobs was under-called. Cal.com has a genuine job system: a Tasker package with an internal tasker, a Redis-backed tasker, and a task processor, plus cron routes. That is real worker and queue behavior, and DevTime only surfaced dependency and test evidence at low confidence. The guard that suppresses fake background-job signals (calendar cleanup crons pretending to be job queues) is suppressing a true positive here.

Admin Permissions was missed entirely. The repo has real admin surfaces, including admin API routes and admin settings layouts. DevTime detected nothing.

Both misses become regression fixtures now: small repo patterns encoding the expected output, so this exact failure can never return silently. Funny detail: two of the oldest fixtures in the suite came from Cal.com-shaped false positives (calendar "subscriptions" being mistaken for billing). Those fixtures fixed overclaiming. Today Cal.com exposed the opposite failure, overcorrection. Same factory, opposite direction.

One more honesty note: during the scan, DevTime printed this on its own:

Framework coverage warning: NestJS controller parsing is not supported in V0;
backend route coverage may be incomplete.
Enter fullscreen mode Exit fullscreen mode

Cal.com's API v2 is NestJS. The tool told me where its own blind spot was before I found it.

Why I Care About This Pattern

Every tool in this space is optimized to sound sure. Summaries, autocomplete, agents: confidence is the product.

But the Cal.com webhook stub is exactly the kind of thing confident tools get wrong. The story "this repo handles Stripe webhooks" is plausible, well-supported by file names, and false. If your AI agent believed it, it might modify payment logic that does not exist in this edition, or trust dedupe behavior that is not there.

The rule I keep coming back to:

No claim without evidence. Weak evidence produces uncertainty, not confidence.

Uncertainty is not the tool failing. Uncertainty is the tool working.

Try It on Your Repo 

pipx install devtime-ei
cd your-repo
dtc init
dtc scan
dtc concepts
Enter fullscreen mode Exit fullscreen mode

There is also a read-only MCP server (dtc mcp start) so coding agents can query the same evidence instead of guessing.

If DevTime gets something wrong on your repository, that is not an embarrassment, it is a contribution. Wrong outputs become fixtures, and fixtures are how this tool earns trust.

Links

GitHub logo Shakargy / devtime

Local-first Engineering Intelligence for software repositories.

DevTime

Local-first Engineering Intelligence for software repositories.

DevTime helps a codebase explain itself from evidence.

It scans code, tests, configs, routes, and decisions to identify supported software concepts, link claims to files, surface uncertainty, and warn about a narrow set of risky changes.

No cloud. No telemetry. No code execution. No AI required.

DevTime terminal demo - install, scan, and explain a repo from evidence

Prefer video? Watch the 2-minute demo: DevTime scans a repo locally, explains concepts from evidence, surfaces uncertainty, catches a risky diff, and shows how a corroborated decision improves understanding.

Try DevTime in 60 seconds

pipx install devtime-ei
dtc demo init
cd devtime-demo-saas
dtc init
dtc scan
dtc concepts
dtc explain "Billing Webhooks"
Enter fullscreen mode Exit fullscreen mode

The PyPI distribution is devtime-ei. The Python package remains devtime, and the CLI command remains dtc. dtc demo init copies a small static example repo into ./devtime-demo-saas so you can try DevTime without cloning this repository.

From source

git
…
Enter fullscreen mode Exit fullscreen mode
View on GitHub

If you run it on a repo and it says something your code cannot back up, open an issue. That is exactly the feedback I am looking for!

Top comments (0)