Case Study: Freeze a Telemetry Catalog Before Generated Code Invents Event Names

Invented telemetry names are a silent product bug, and you should reject them before any model writes a comment. Cheap code generation makes extra track() calls feel free, then your dashboards quietly fork into two dialects. This case study walks one small checkout service from catalog freeze through a scanner, a near-miss report, and a review note. You keep the gate deterministic; you ask a model only to explain names that already failed the catalog.

Background

You maintain a tiny Python checkout service with three real product events and a messy habit of copy-paste instrumentation. An agent helping on a refund path will often emit order_refunded_v2 because that string sounds newer than order.refunded. Warehouse queries then split, alerts miss the new name, and nobody notices until a weekly revenue review. The failure is not missing AI. The failure is accepting unpublished names as if they were a product decision.

You do not need a platform team or a paid observability suite for the first version. You need one frozen YAML file, an AST walk over track( and emit(, and a CI rule that treats unknown strings as build failures. A model can still help later, but only after the catalog has already said no.

Goal

You want a gate that a teammate can run in one command and understand in one screen. The worked example below is a synthetic service, not a production claim, and every count comes from the fixtures in this article. Success looks like four outcomes you can replay on a laptop:

• Exact catalog hits pass without involving a model.
• Unknown literals fail the scan with file, line, and symbol.
• Near-miss names get a separate report so typos do not look like new features.
• Optional prose explains why a near-miss is dangerous, citing the catalog path, never inventing a replacement event.

The frozen catalog

You treat event names as an API. If a string is missing from events.yaml, it is a bug, not a suggestion. Keep values dotted, past-tense, and boring so generated code cannot hide a product change inside a creative synonym.

# events.yaml
version: 1
events:
  - name: checkout.started
    owner: payments
    pii: false
  - name: order.placed
    owner: payments
    pii: false
  - name: order.refunded
    owner: payments
    pii: false

You also freeze the call shapes you will accept, because agents love adding capture(), log_event(), and analytics.send() as “helpful” wrappers. For this case you allow only track and emit with a string literal as the first argument. Dynamic names are out of scope on purpose; they belong in a later RFC, not in an emergency patch.

Implementation

The scanner is ordinary Python. You parse each file, walk ast.Call nodes, and collect first-argument constants. That keeps the gate cheap, reviewable, and independent of any vendor prompt.

# scan_events.py
from __future__ import annotations

import argparse
import ast
import pathlib
import sys
from difflib import get_close_matches

import yaml

ALLOWED = {"track", "emit"}


def load_catalog(path: pathlib.Path) -> set[str]:
    data = yaml.safe_load(path.read_text())
    return {row["name"] for row in data["events"]}


class EventVisitor(ast.NodeVisitor):
    def __init__(self, rel: str) -> None:
        self.rel = rel
        self.hits: list[tuple[int, str, str]] = []

    def visit_Call(self, node: ast.Call) -> None:
        name = None
        if isinstance(node.func, ast.Name):
            name = node.func.id
        elif isinstance(node.func, ast.Attribute):
            name = node.func.attr
        if name in ALLOWED and node.args:
            arg = node.args[0]
            if isinstance(arg, ast.Constant) and isinstance(arg.value, str):
                self.hits.append((arg.lineno, name, arg.value))
        self.generic_visit(node)


def scan(root: pathlib.Path, catalog: set[str]) -> tuple[list[str], list[str]]:
    unknown: list[str] = []
    near: list[str] = []
    for path in root.rglob("*.py"):
        if path.name == "scan_events.py":
            continue
        tree = ast.parse(path.read_text(), filename=str(path))
        visitor = EventVisitor(str(path.relative_to(root)))
        visitor.visit(tree)
        for lineno, func, event in visitor.hits:
            loc = f"{visitor.rel}:{lineno} {func}({event!r})"
            if event in catalog:
                continue
            unknown.append(loc)
            close = get_close_matches(event, sorted(catalog), n=1, cutoff=0.72)
            if close:
                near.append(f"{loc} ~ {close[0]}")
    return unknown, near


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--root", type=pathlib.Path, default=pathlib.Path("."))
    parser.add_argument("--catalog", type=pathlib.Path, default=pathlib.Path("events.yaml"))
    args = parser.parse_args()
    catalog = load_catalog(args.catalog)
    unknown, near = scan(args.root, catalog)
    if not unknown:
        print("ok: all telemetry names are in the catalog")
        return 0
    print("unknown telemetry names:")
    for row in unknown:
        print(f"  - {row}")
    if near:
        print("near misses (typo vs new product event):")
        for row in near:
            print(f"  - {row}")
    return 1


if __name__ == "__main__":
    sys.exit(main())

You then add a fixture module that mimics a generated refund patch. Leave the three legal events in place so the happy path stays visible in the same file.

# app/checkout.py
def track(event: str, **payload: object) -> None:
    print(event, payload)


def emit(event: str, **payload: object) -> None:
    print(event, payload)


def start_checkout(user_id: str) -> None:
    track("checkout.started", user_id=user_id)


def place_order(order_id: str) -> None:
    emit("order.placed", order_id=order_id)


def refund_order(order_id: str) -> None:
    # Invented by a helpful agent during a refund patch.
    track("order_refunded_v2", order_id=order_id)
    emit("order.refunded", order_id=order_id)

Run the gate the same way CI will run it. You should see a non-zero exit and one near-miss against order.refunded.

pip install pyyaml
python scan_events.py --root . --catalog events.yaml
echo $?

Optional explanation after the fail

Only when the scanner already failed do you ship a compact bundle to a model. The bundle is the unknown row, the closest catalog name, and a hard instruction: do not propose a new event. That split matters because models are fluent at minting plausible product language, which is exactly the failure mode you are trying to stop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already have MonkeyCode's free model access, you can send that bundle through it without putting a model in the blocking path. The free server option is useful when you want a scheduled scan of main in addition to pull-request CI, not as a replacement for the AST check.

# explain_near_miss.py  (proposal: run only after scan_events.py exits 1)
PROMPT = """You are reviewing a CI failure, not designing analytics.
Unknown event: {unknown}
Closest catalog name: {closest}
Write four sentences for the pull request.
Cite the catalog name. Do not invent a replacement event.
Do not suggest renaming the catalog to match the patch.
"""

Label the snippet as a proposal. You still decide the HTTP client, the secret store, and the comment poster for your own forge. The important contract is the order of operations: catalog first, similarity second, prose last.

Results from the fixtures

Re-running the scanner against the three functions in app/checkout.py produces a stable, boring report. You should treat these numbers as fixture results, not as a production benchmark.

Check	Count in this repo	Gate action
Catalog events	3	Source of truth
Literal track / emit calls	4	Inspected
Exact catalog hits	3	Pass
Unknown literals	1 (order_refunded_v2)	Fail CI
Near misses at cutoff 0.72	1 (order.refunded)	Attach explanation
Model calls on a clean tree	0	Skip

You learn two operational facts from that table. First, the legal refund event was already present, so the invented alias was not filling a gap. Second, the near-miss line stops a reviewer from arguing about taste; the catalog already chose order.refunded.

What you schedule on a free server

Pull-request CI catches new patches. A nightly job still matters because someone will commit generated code with --no-verify on a Friday. Keep the job dumb: clone, install pyyaml, run the scanner, and page only on unknown names. You do not need a GPU for an AST walk, and you should not wait for a model if the tree is clean.

# proposal: cron entry on a small always-on box
*/30 * * * * cd /srv/event-gate && git pull --ff-only && python scan_events.py --root ./service --catalog ./events.yaml

If that box is MonkeyCode's free server option, use it as a scheduler and an artifact host for the last failing report. Do not move the allow/deny decision into a chat transcript, because transcripts are not diffs you can replay.

Limitations

This design is intentionally narrow, and you should read the gaps before you copy it into a monorepo.

• ast sees string literals only. track(EVENT_REFUND) and track(f"order.{action}") are invisible.
• Attribute aliases such as analytics.track are collected by function name, which can false-positive on unrelated emit helpers.
• get_close_matches is not product sense. order.returned may sit near order.refunded and still be a real new event.
• YAML ownership fields are documentation. The scanner does not page the listed owner.
• A model asked to “be helpful” will still try to mint order.refunded.v2 unless the prompt forbids new names.
• The fixture refund function emits both the invented name and the legal name, which is kinder than most generated patches.

Who should not use this approach

You should not freeze a catalog if your product truly requires runtime-defined event names, such as customer-specific workflow steps. You should not point this scanner at generated protobuf stubs and expect it to understand enum aliases. You should not hire a model to invent the first catalog either; that just launders guesses into YAML. If you lack authority to reject a pull request, the gate becomes a comment bot, and comment bots do not protect dashboards.

Lessons learned

Cheap generation does not make unpublished telemetry cheaper to store. It makes extra names cheaper to type, which is the opposite of a data contract. You protect the contract with a file you can diff, a walker you can unit-test, and a model that is not allowed to extend the vocabulary. The interesting review question is no longer “does this name sound clear?” The question is “did we already name this fact?”

Keep the human decision at catalog change time, not at patch time. When someone truly needs order.refund.failed, they add a YAML row with an owner, they ship the scanner update in the same pull request, and they only then regenerate code. That sequence is slower than letting an agent improvise, and that slowness is the point.

If you want a small next step, add one unit test that feeds scan_events.py a temporary directory containing a single invented literal. After that works, park the same command on a free server and leave the model behind a failing exit code. MonkeyCode’s free model access is enough for the optional review paragraph when you already trust the catalog more than the prompt.