I was running a planned refactor across a medium-sized TypeScript repository using an AI assistant to suggest and rewrite function names and local variable names. The assistant produced plausible changes per file: concise names, clearer intent, and fewer comments. The problem appeared benign until integration tests started failing intermittently; the AI had renamed a loop accumulator in one file but not its counterpart in another module. At first the mismatch was invisible because each file compiled and unit tests passed in isolation. The CI pipeline, which runs end-to-end scenarios, exposed the failure where serialized state used across modules depended on a naming convention the AI had subtly changed. I had relied on the assistant’s suggestions as refactor drafts and assumed renames would be mechanically consistent across the codebase.
How the inconsistency manifested in practice
In our workflow the model suggested edits file-by-file, and we accepted many small hunks. The AI would propose a new variable name in module A (for example, changing currentIndex to idx) and a slightly different name in module B (current_idx or i) even when both variables were conceptually the same and passed through shared interfaces. Because type hints and tests were local, TypeScript sometimes inferred types compatibly and the editor showed no red flags.
We used the assistant interactively through a chat session, iterating on suggestions. The multi-turn conversation made it feel like a single cohesive operation, but the model’s lack of global file awareness meant it optimized each suggestion locally. The inconsistency compounded when small behavioral differences—like off-by-one index differences in renamed loops—were introduced alongside the naming drift, producing subtle logic regressions.
Why the failure was subtle and easy to miss
The subtlety comes from several small behaviors adding up: the model treats files as separate contexts, token limits reduce global context, and refactor suggestions prioritize human-readable names over strict structural matching. Those optimization choices are reasonable in isolation but are dangerous during cross-file renames where exact identity matters. A codebase relying on stringly-typed keys, serialized payloads, or reflection is particularly vulnerable.
I cross-checked suspicious changes with a deep research workflow to verify references, but the research tool surfaces matches rather than enforcing transformations. That meant the burden remained on engineers to detect inconsistent renames. The hardest cases were when the new names still passed linters and local tests, creating false confidence that the refactor was safe.
Practical mitigations and verification steps
Treat AI refactors as guided diffs, not automated global transforms. Before accepting multi-file renames, generate a reference map of old-to-new identifiers and run automated checks: grep the repository for textual matches, use the language server to verify symbol-level renames, and run full integration tests. Prefer structural refactors via the editor or dedicated rename tools that operate on the AST instead of accepting per-file suggestions blind.
Also, add small contract tests that validate serialized schemas and cross-module APIs; these catch naming drift in payloads. Finally, keep human review focused on identity-preserving changes and be explicit in prompts—ask the assistant to propose consistent global renames and then verify mechanically. AI can speed refactors, but its local optimization behavior can quietly introduce cross-file mismatches unless you verify at repository scale.
Top comments (0)