We were refactoring a small service and leaned on a code-generation model to speed up cross-file renames. In response to prompts, the model suggested updated function names, argument names, and even exports across several files. At first glance those diffs looked reasonable — the variable names were more descriptive — so we merged them and moved on, using the model interactively via a chat interface during the work. The problem didn’t show up immediately. Unit tests still passed, the linter complained in a few places, and CI ran green. Production traffic later exposed a handful of endpoints returning 500s. The tracebacks were simple: undefined is not a function, or module has no exported member. The root cause was inconsistent naming introduced by the model: a variable called userId in one file, user_id in another, and userIdentifier in a generated helper. Small naming differences cascaded into runtime mismatches when the wrong symbol was imported.
How it surfaced during a refactor
Our workflow had multiple files touched in one PR: API handlers, middlewares, and a shared util module. We prompted the model to produce changes per-file, rather than asking it to refactor the whole module graph. That made the model produce locally consistent suggestions for each file, but without global coordination; imports and exports were adjusted in some places and left untouched in others. We also generated asset keys for front-end references and image handling with the AI Image Generator-driven pipeline, which compounded naming drift between backend keys and front-end assets.
The error only became obvious at integration points: when an API handler imported a renamed helper but the shared module still exposed the old name. Our stack traces pointed to the import lines but not the semantic mismatch. Debugging required switching between files to see that the symbol names didn’t line up — exactly the sort of cross-file context the model was loose about when it suggested edits one file at a time.
Why it was subtle and easy to miss
Two model behaviors made this easy to overlook. First, the model confidently proposes names that look plausible, which tempts developers to accept suggestions without deep verification. Second, multi-turn or multi-file sessions can lose implicit project-level constraints: the model treats each prompt as a fresh context unless you explicitly include the other files. We documented the mismatch and used a deep research approach to cross-check naming conventions and exports, but only after the bug surfaced.
These issues are subtle because unit tests often mock the same module names the refactor changed, and linters catch style but not semantic divergence across modules. If your build system tolerates unresolved imports until runtime (dynamic requires, permissive bundlers), the breakage can slip past CI entirely and surface only under specific request patterns.
Mitigations and practical takeaways
Treat AI-generated refactors as a draft: use the model for suggestions but run deterministic, tool-assisted refactors (IDE rename, codemods) for global changes. Add stricter compile-time checks, type systems, and export/import validation into CI so cross-file mismatches fail early. If you use the model interactively, include all relevant files in the prompt or explicitly assert naming conventions to reduce drift.
Finally, update your PR checklist: require a full integration test run, enforce export/import lints, and review model suggestions with extra scrutiny on public APIs. Small naming divergences are low-severity in isolation, but when a model propagates them across many files they become a reliability hazard — and the fix is procedural rather than magical.
Top comments (0)