A static analysis rule written from a spec is a hypothesis

Last week I shipped a tool that predicts which ComfyUI custom nodes will break on your next git pull. It answered one question: does every name a pack imports still exist upstream? That question has a clean answer, and the tool got it right.

It was also the wrong question, or at least only half of it.

Import success is not load success

ComfyUI's comfy.* modules are internal. No deprecation policy, no __all__, no shim. Packs import from them anyway because there is no other way to hook the sampler or patch a model. So when a refactor lands, packs die:

ImportError: cannot import name 'precompute_freqs_cis' from 'comfy.ldm.lightricks.model'

Static analysis handles that well. Build the set of names bound at module scope from the AST, diff it against what the pack references, done.

But scroll the 2026 issue tracker and the loudest failures aren't ImportErrors at all:

TypeError: calculate_weight() got an unexpected keyword argument 'intermediate_dtype'
TypeError: WanAttentionBlock.forward() got an unexpected keyword argument 'context_img_len'
TypeError: patched_forward_orig() got an unexpected keyword argument 'timestep_zero_index'

Every name resolves. The symbol is right there. The parameter list moved, and either the pack calls it with the old shape, or the pack replaced the function with its own copy and core now passes an argument that copy never learned about.

So I wrote the obvious rule: parse the target module at whatever git ref you care about, pull the real parameter list, and try to bind every call the pack makes. A call that can't bind is a break, because it raises TypeError the moment it runs.

That rule is a hypothesis. I nearly shipped it as a fact.

Measuring it on 20 real packs

Before release I cloned the 20 most popular ComfyUI node packs, ran the new rules over all of them against origin/master, and hand-verified every hit against both the pack source and ComfyUI's git history. 1,273 Python files. 1,116 call sites into comfy.*. 10 monkeypatches.

The rule as specified was wrong in two ways, and both would have failed working packs' CI.

False positive one: shadowed names

from comfy.lora import calculate_weight

def calculate_weight(x):     # the pack's own
    ...

calculate_weight(x)          # not the one you imported

My rule resolved that call against upstream and reported a hard break on code that is completely fine. Any rebinding does it: a local def, a function parameter, a loop target, a walrus, a later import of the same name.

The fix is a conservative file-wide shadow set, but the interesting part is that I deliberately did not apply the same strictness to the import checks. Presence checking can afford to be loose about shadowing, because its false positive is a warning somebody glances at and dismisses. A call-binding check cannot, because its false positive is a red build. Same codebase, two different tolerances, chosen by what the wrong answer costs.

False positive two: the version shim, which punishes the careful

This one is the reason I'm writing this post.

PREFETCH_CLEANUP_TAKES_MODULE = hasattr(comfy.model_prefetch, "GRAPH_MODULES")
...
if PREFETCH_CLEANUP_TAKES_MODULE:
    comfy.model_prefetch.cleanup_prefetched_modules(prefetched_module, comfy_modules)
else:
    comfy.model_prefetch.cleanup_prefetched_modules(comfy_modules)

That pack probes ComfyUI at runtime and calls the correct arity for whichever version you have. It is doing compatibility properly.

At any given ref, exactly one of those branches binds and the other is dead code. My rule saw the dead branch, couldn't bind it, and graded the pack WILL BREAK.

Think about who that hurts. Not sloppy packs. The rule fired hardest on the pack that had gone to the trouble of supporting multiple ComfyUI versions. A lint that taxes the careful and ignores the careless is worse than no lint.

The fix: if a sibling call to the same function in the same file binds, the failing one is a shim. Report it, never fail the build. I also made except TypeError around a call soften it, the same way except ImportError already softens an import, since that's the other explicit way people say "I know the signature might not match here."

What survived

Two findings across 1,116 call sites. Both true. Eighteen of twenty packs completely silent.

The interesting one is in ComfyUI-Easy-Use, a pack with 2.7k stars. Its BrushNet path calls:

comfy.ops.pick_operations(..., scaled_fp8=model.model.model_config.scaled_fp8)

scaled_fp8 was removed from that function in ComfyUI commit 43071e3de (PR #11000, Dec 2025). Last release that accepts it is v0.3.77. First that doesn't is v0.4.0. On anything newer, that call raises TypeError.

Then I went to file it and found issue #991, opened four months ago, in Chinese, still open, with a user-discovered workaround: delete the argument. Nobody had connected it to the upstream commit that caused it, and nobody had noted the version boundary, which matters because if the pack still supports ComfyUI below v0.4.0 then deleting the argument breaks the older path instead of fixing anything.

That reframed the tool for me. The value wasn't finding an unknown bug. It was attaching a commit, a PR and a release boundary to a known one that had been sitting there for four months as "just delete this line."

The part worth keeping

A static analysis rule written from a specification is a hypothesis about real code. Mine passed every unit test I wrote for it while being wrong about two entire categories of correct code, because I had written the tests from the same wrong mental model as the rule.

Twenty real repositories cost an afternoon and killed both. Measure the hit rate, not just the correctness, and pay attention to who your false positives land on. If they land on the people doing it right, the rule is a tax, not a signal.

The tool is comfy-import-guard, MIT, pip install comfy-import-guard. Zero third-party dependencies, because it has to load inside a ComfyUI whose other packs are already broken.