sunnydachs

Posted on Sep 18

The tracking plan said "implemented". The dashboard says otherwise. I built a CLI that detects plan↔code drift using only AST.

#python #cli #analytics #opensource

"Did shipping auth tracking make it into the release?"
"Yep, it's in."

Two weeks later, someone opens the dashboard and says "…this doesn't look right."

This is about plan-drift, a CLI I built that detects exactly that gap — the silent drift between your analytics tracking plan (what you intended to measure) and the actual implementation — without using an LLM at all.

https://github.com/sunnydachs/plan-drift

What it does

You give it a repository and a tracking plan (a JSON file). It finds four kinds of drift:

UNEXPECTED EVENT — implemented in code, but not in the tracking plan
UNIMPLEMENTED EVENT — in the plan, but no track() call found
PROPERTY MISMATCH — event name matches, but its properties diverge (undeclared props, missing required)
DYNAMIC — the event name or properties can't be resolved statically (manual check needed)

# scan the current repo against your plan (read-only)
plan-drift --plan tracking-plan.json

# another path, JSON output
plan-drift --plan tracking-plan.json ./src --json

Output looks like this:

plan-drift — scanned /home/dev/myproject
  tracking plan events: 4 | track() calls found: 6 (dynamic: 1)
app/events.py:12  [!] PROPERTY MISMATCH
    event 'Signed Up': undeclared property 'campaign' found in code but not in plan
app/analytics.py:34  [+] UNEXPECTED EVENT
    'Add To Cart' implemented (1 call) but not in the tracking plan
app/analytics.py:56  [ ] UNIMPLEMENTED EVENT
    'Checkout Started' declared in plan but no track() call found

"The event shipped but the properties are off" is invisible to IDEs and static analysis. Tracking code is rarely covered by tests. So nobody notices until the numbers look wrong.

Why deliberately not use an LLM

This was the core design decision. LLMs could parse ambiguous tracking code with more "understanding." But I built it on nothing but Python's standard ast (Abstract Syntax Tree). Three reasons:

Deterministic. Same input → same report, always. You can't put a tool in CI whose verdict changes run to run.
Read-only and safe. It never generates or edits code. It parses the syntax tree and compares.
Zero dependencies. Python 3.11+ is enough. pip install git+... and you're done.

Tracking calls like analytics.track(...) are statically analyzable — an LLM here would add cost, latency, and nondeterminism with no accuracy gain.

This is one answer to a question I keep coming back to when automating: how much should AI handle? Deterministic work deserves deterministic tools.

The design core: check both directions

The thing I cared most about was checking plan↔implementation in both directions:

An implemented event NOT in the plan is a dangerous signal — could be an unexpected addition, could be a misconfiguration
A planned event with no implementation is also a problem — the classic "the plan got written, nobody shipped it"

So the tool cross-references and enumerates every mismatch. Test files (tests.py, test_*.py) are excluded, because fixture events showing up as "unplanned implementations" would be pure noise.

Where it fits

Marketing team writes the plan → devs instantly see how far implementation has come
Renaming an event or dropping a property without updating the plan → caught immediately
Wire into CI to warn on critical drift

The "plan exists but implementation never got updated" problem I've seen at workplaces — this gets you most of the way there.

Honest limitations

Dynamic events are reported, not resolved — track(event_name, props) shows up as DYNAMIC for manual review.
Python only for now (.py). JS needs a different parser — planned.
Only key presence is checked, not values — if the plan says plan: string and the code sends an int, that's a future improvement.

These are documented on the roadmap. The current version prioritizes correctness and low false positives over coverage.

Wrap-up

Drift between planning and implementation quietly corrupts your dashboards. plan-drift fights it with a read-only, fully deterministic, LLM-free design — accuracy and reproducibility first.

Happy to hear any feedback — whether you're deploying it in-house or just trying it out. It's on GitHub.